Hello All,
I wanted to understand the relationship between number of cores and loading in Vertica.
I remember there is some dependency on number of cores while inserting data into vertica. whether it is the INSERT statement or COPY statements.
Any limitations / Any dependency / any points to remember
Thanks
Relationship between Cores and loading
+2+2Hi Navin,
The only relationship here, generally speaking, is that COPY will parse up to "#cores" files at once, when it is loading files from local disk on the server. (Parsing is often CPU-bound, so we hope this is a reasonable limit.) If you give us a list of 100 files, but you only have 24 CPU cores, then we will process the first 24 files; then, when the first of those files finishes parsing, we'll pick up the 25th, then the 26th, etc.
(Many versions ago, Vertica's COPY used to simply try to load all files at once. This worked ok for smaller numbers of files, but used rather a lot of system resources when loading tens or hundreds of thousands of files.)
Say you have 24 cores and are loading 25 equal-sized files. You might think that this would take twice as long as loading 24 files, because we'd load 24 files at first, then have a single thread loading the last file. In Vertica 6, this was an issue. As of Vertica 7, when we notice that a CPU core has no files left to parse, we will assign that core to help parse an existing file. So we should generally be efficient even in this case.
Adam
The only relationship here, generally speaking, is that COPY will parse up to "#cores" files at once, when it is loading files from local disk on the server. (Parsing is often CPU-bound, so we hope this is a reasonable limit.) If you give us a list of 100 files, but you only have 24 CPU cores, then we will process the first 24 files; then, when the first of those files finishes parsing, we'll pick up the 25th, then the 26th, etc.
(Many versions ago, Vertica's COPY used to simply try to load all files at once. This worked ok for smaller numbers of files, but used rather a lot of system resources when loading tens or hundreds of thousands of files.)
Say you have 24 cores and are loading 25 equal-sized files. You might think that this would take twice as long as loading 24 files, because we'd load 24 files at first, then have a single thread loading the last file. In Vertica 6, this was an issue. As of Vertica 7, when we notice that a CPU core has no files left to parse, we will assign that core to help parse an existing file. So we should generally be efficient even in this case.
Adam
Hi Adam,
We are talking loading into Vertica, I want utilize same thread ask this question.
Could you please help on this
I want find out some load analysis to monitor the health of the system, like system is reaching its max load capacity and writes per seconds , This is very critical for us do some capacity planning.
thank you
Sai
need some answer please
+2Thanks for your interest -- if you need prompt support, please contact Vertica Support at my.vertica.com. These are public forums; you may or may not get an answer, and you are very unlikely to get a response within half an hour.
If you're looking for a quick response, I'll give you a quick answer -- Vertica's Management Console provides you with these tools.
If you're looking for a quick response, I'll give you a quick answer -- Vertica's Management Console provides you with these tools.
Sign up
Already have an account? Login
Welcome to the Rocket Forum!
Please log in or register:
Employee Login | Registration Member Login | RegistrationEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.
77 4th Avenue
Waltham, MA 02451 USA
