Huge files in hadoop: how to store metadata? -


i have use case upload tera-bytes of text files sequences files on hdfs.

these text files have several layouts ranging 32 62 columns (metadata).

what way upload these files along metadata:

  1. creating key, value class per text file layout , use create , upload sequence files ?

  2. create sequencefile.metadata header in each file being uploaded sequence file individually ?

any inputs appreciated !

thanks

the simplest thing make keys , values of sequencefiles text. pick meaningful field data make key, data value text. sequencefiles designed storing key/value pairs, if that's not data don't use sequencefile. upload unprocessed text files , input hadoop.

for best performance, not make each file terabytes in size. map stage of hadoop runs 1 job per input file. want have more files have cpu cores in hadoop cluster. otherwise have 1 cpu doing 1 tb of work , lot of idle cpus. file size 64-128mb, best results should measure yourself.


Comments

Post a Comment

Popular posts from this blog

apache - Add omitted ? to URLs -

i working on website cms reads page name of query string, e.g. http://exmaple.com/?pagename . trying create rewrite rule rewrite ? . so, user types in /pagename , taken /?pagename try this: rewriteengine on rewritecond %{request_filename} !-d rewritecond %{request_filename} !-f rewriterule ^/?(.*) index.php?$1 [l]
Read more

redirect - bbPress Forum - rewrite to wwww.mysite prohibits login -

i using bbpress installation without wordpress integration. the redirect of rewriteengine @ forum site works: rewriteengine on rewritecond %{http_host} ^forum.mysite.com$ rewriterule (.*)$ http://www.forum.mysite.com/$1 [r=301,l] the problem login pppress. bbpress login not recognize: http://www.forum.mysite/bb-login.php . it redirects to: http://forum.mysite.com/bb-login.php . i tried permalink settings none , name based, same issue. does uses redirect in bbpress or how can eliminate "double" content , redirect permanently www.? i redirected www.forum.mysite.com forum.mysite.com , achieved eliminate "double" content.
Read more

php - How can I stop spam on my custom forum/blog? -

so have custom built forum & blog system of late has been dealing lot of spam. if wordpress use akismet, if different common platform i'm sure i'd find plugin. there kind of static class can download this? using php. i'd still go akismet, if it. uses outside of wordpress, may have pay fee it, depending on use -- check terms , conditions -- it's option , easy implement in php using api. use api key wordpress. com account access. basically, grab whichever php client library takes fancy -- use alex potsides' php5 library -- plug in key, , it's handful of lines of code. here's bare bones of validation straight 1 of live sites: ... if ($akismet) { $akismet->setcommentauthor($name); $akismet->setcommentauthoremail($session->userinfo["email"]); $akismet->setcommentauthorurl(""); $akismet->setcommentcontent($sentence);...
Read more