python - What is the most performant way to store a list of Tuples in App-Engine? -
when storing , retrieving datastore entity contains list of tuples efficient way of storing list?
when have encountered problem tuples key value pairs, datetime , sample results, (x, y) coordinates.
number of tuples variable , ranges 1 few hundred.
the entity containing these tuples, need referenced quickly/cheaply, , tuple values not need indexed.
i have had problem few times, , have solved number of different ways.
method 1:
convert tuple values string , concatenate them delimiter.
def putentity(entity, tuples): entity.tuples = ['_'.join(tuple) tuple in tuples] entity.put()
advantages: results readable in datastore viewer, fetched in 1 get. disadvantages: potential precision loss, programmer required deserialize/serialize, more bytes required store data in string format.
method 2:
store each tuple value in list , zip / unzip tuple.
def putentity(entity, tuples): entity.keys = [tuple[0] tuple in tuples] entity.values = [tuple[1] tuple in tuples] entity.put()
advantages: no loss of precision, confusing still possible view data in datastore viewer, able enforce types, fetched in 1 get.
disadvantage: programmer needs zip / unzip tuples or maintain order in lists.
method 3:
serialize list of tuples in manor json, pickle, protocol buffers , store in blob or text property.
advantages: usable objects, , more complex objects, less risk of bug miss matching tuple values.
disadvantages: blob store access requires , additional fetch?, can not view data in datastore viewer.
method 4:
store tuples in entity , keep list of keys.
advantages: more obvious architecture. if entity view, no longer need keep 2 copies of tuple data.
disadvantages: 2 fetches required 1 entity , key list , 1 tuples.
i wondering if knows 1 performs best , if there way haven't thought about?
thanks, jim
i use method 3. blobstore may require fetch, db.blobproperty not. objects important comes out of storage put in use pickleproperty (which can found in tipfy, , other utility libraries).
for objects need state stored wrote jsonproperty function works pickleproperty (but uses simplejson, obviously).
for me getting data in single fetch, , being idiot-proof, more important cpu performance (in app engine). according google i/o talk on appstats, trip datastore going more expensive bit of local parsing.
Comments
Post a Comment