Dtjytyk

Weaponising the Grasp-at-a-Distance spell

Why are current probes so expensive?

IC on Digikey is 5x more expensive than board containing same IC on Alibaba: How?

"Destructive power" carried by a B-52?

What did Turing mean when saying that "machines cannot give rise to surprises" is due to a fallacy?

What helicopter has the most rotor blades?

.bashrc alias for a command with fixed second parameter

Random body shuffle every night—can we still function?

Keep at all times, the minus sign above aligned with minus sign below

What is "Lambda" in Heston's original paper on stochastic volatility models?

Why did Bronn offer to be Tyrion Lannister's champion in trial by combat?

How do I find my Spellcasting Ability for my D&D character?

How does TikZ render an arc?

Plotting a Maclaurin series

Why do the Z-fighters hide their power?

How to resize main filesystem

Marquee sign letters

How do you write "wild blueberries flavored"?

The test team as an enemy of development? And how can this be avoided?

Is there a verb for listening stealthily?

What is the proper term for etching or digging of wall to hide conduit of cables

Flight departed from the gate 5 min before scheduled departure time. Refund options

Does a random sequence of vectors span a Hilbert space?

Shimano 105 brifters (5800) and Avid BB5 compatibility

What is a good way to store processed CSV data to train model in Python?

1

$begingroup$

I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.

Some options appear to be:

• HDFS


• HDF5


• HDFS3


• PyArrow

python keras dataset csv serialisation

share|improve this question

edited Mar 26 at 10:30

Vaalizaadeh

7,65562265

asked Mar 26 at 9:52

B SevenB Seven

22218

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  When I want to got 5 mts in distance, I would rather walk than to take a car.
  $endgroup$
  – Kiritee Gak
  Mar 26 at 10:23
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
  $endgroup$
  – SoK
  Mar 26 at 10:32
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Just leave it as CSV you don't need to do anything
  $endgroup$
  – arhwerhwe
  Mar 26 at 11:27
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
  $endgroup$
  – n1tk
  Mar 26 at 18:24
  

  
  
  
  

add a comment | 

1

$begingroup$

I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.

Some options appear to be:

• HDFS


• HDF5


• HDFS3


• PyArrow

python keras dataset csv serialisation

share|improve this question

edited Mar 26 at 10:30

Vaalizaadeh

7,65562265

asked Mar 26 at 9:52

B SevenB Seven

22218

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  When I want to got 5 mts in distance, I would rather walk than to take a car.
  $endgroup$
  – Kiritee Gak
  Mar 26 at 10:23
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  I think HDF5 is very good for you, your data size is small, I am working on h5 files it's fast.
  $endgroup$
  – SoK
  Mar 26 at 10:32
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Just leave it as CSV you don't need to do anything
  $endgroup$
  – arhwerhwe
  Mar 26 at 11:27
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Why not dump the dataframe to_pickle ? Easy, low memory, compression supported and fast loading without specifying columns or other parameters ...
  $endgroup$
  – n1tk
  Mar 26 at 18:24
  

  
  
  
  

add a comment | 

$begingroup$

I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.

Some options appear to be:

• HDFS


• HDF5


• HDFS3


• PyArrow

python keras dataset csv serialisation

share|improve this question

edited Mar 26 at 10:30

Vaalizaadeh

7,65562265

asked Mar 26 at 9:52

B SevenB Seven

22218

$endgroup$

share|improve this question

edited Mar 26 at 10:30

Vaalizaadeh

7,65562265

asked Mar 26 at 9:52

B SevenB Seven

22218

share|improve this question

edited Mar 26 at 10:30

Vaalizaadeh

7,65562265

asked Mar 26 at 9:52

B SevenB Seven

22218

edited Mar 26 at 10:30

Vaalizaadeh

7,65562265

edited Mar 26 at 10:30

Vaalizaadeh

7,65562265

asked Mar 26 at 9:52

B SevenB Seven

22218

asked Mar 26 at 9:52

B SevenB Seven

22218

3 Answers
3

active

oldest

votes

4

$begingroup$

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.

Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.

share|improve this answer

answered Mar 26 at 10:11

Shamit VermaShamit Verma

1,6541414

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
  $endgroup$
  – Bilkokuya
  Mar 26 at 14:21
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
  $endgroup$
  – B Seven
  Mar 26 at 17:05
  

  
  
  
  

add a comment | 

3

$begingroup$

You can find a nice benchmark for every approach in here.

share|improve this answer

answered Mar 26 at 11:15

Francesco PegoraroFrancesco Pegoraro

624119

$endgroup$

add a comment | 

1

$begingroup$

Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.

share|improve this answer

answered Mar 26 at 10:30

VaalizaadehVaalizaadeh

7,65562265

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48008%2fwhat-is-a-good-way-to-store-processed-csv-data-to-train-model-in-python%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

4

$begingroup$

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.

Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.

share|improve this answer

answered Mar 26 at 10:11

Shamit VermaShamit Verma

1,6541414

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
  $endgroup$
  – Bilkokuya
  Mar 26 at 14:21
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
  $endgroup$
  – B Seven
  Mar 26 at 17:05
  

  
  
  
  

add a comment | 

4

$begingroup$

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.

Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.

share|improve this answer

answered Mar 26 at 10:11

Shamit VermaShamit Verma

1,6541414

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  +1 Always profile first. Unless OP has evidence that reading from the data is causing the major bottleneck - they shouldn't be investing resources in optimising it.
  $endgroup$
  – Bilkokuya
  Mar 26 at 14:21
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  That's a good point. I should find out how long it takes. Also, I can see that converting from CSV to DataFrame could take time as well...
  $endgroup$
  – B Seven
  Mar 26 at 17:05
  

  
  
  
  

add a comment | 

$begingroup$

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.

Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.

share|improve this answer

answered Mar 26 at 10:11

Shamit VermaShamit Verma

1,6541414

$endgroup$

share|improve this answer

answered Mar 26 at 10:11

Shamit VermaShamit Verma

1,6541414

answered Mar 26 at 10:11

Shamit VermaShamit Verma

1,6541414

answered Mar 26 at 10:11

Shamit VermaShamit Verma

1,6541414

3

$begingroup$

You can find a nice benchmark for every approach in here.

share|improve this answer

answered Mar 26 at 11:15

Francesco PegoraroFrancesco Pegoraro

624119

$endgroup$

add a comment | 

3

$begingroup$

You can find a nice benchmark for every approach in here.

share|improve this answer

answered Mar 26 at 11:15

Francesco PegoraroFrancesco Pegoraro

624119

$endgroup$

add a comment | 

$begingroup$

You can find a nice benchmark for every approach in here.

share|improve this answer

answered Mar 26 at 11:15

Francesco PegoraroFrancesco Pegoraro

624119

$endgroup$

share|improve this answer

answered Mar 26 at 11:15

Francesco PegoraroFrancesco Pegoraro

624119

answered Mar 26 at 11:15

Francesco PegoraroFrancesco Pegoraro

624119

answered Mar 26 at 11:15

Francesco PegoraroFrancesco Pegoraro

624119

1

$begingroup$

Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.

share|improve this answer

answered Mar 26 at 10:30

VaalizaadehVaalizaadeh

7,65562265

$endgroup$

add a comment | 

1

$begingroup$

Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.

share|improve this answer

answered Mar 26 at 10:30

VaalizaadehVaalizaadeh

7,65562265

$endgroup$

add a comment | 

$begingroup$

Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.

share|improve this answer

answered Mar 26 at 10:30

VaalizaadehVaalizaadeh

7,65562265

$endgroup$

share|improve this answer

answered Mar 26 at 10:30

VaalizaadehVaalizaadeh

7,65562265

answered Mar 26 at 10:30

VaalizaadehVaalizaadeh

7,65562265

answered Mar 26 at 10:30

VaalizaadehVaalizaadeh

7,65562265