python - Scrapy: RSS control pub_date -


i'm doing rss spider. how do controlling last crawl date?

right thinking this:

  • put in control file last pub_date have crawled.
  • then when crawl starts, checks last pub_date against new pub_dates. if there new items, start crawling, if not, nothing.

how else resolve this?

i store data in database (including last crawl date , post dates) , take dates need database.


Comments

Post a Comment

Popular posts from this blog

jQuery clickable div with working mailto link inside -

i have div want clickable need mailto link inside div still work too. when hover on mailto link mailto appears @ bottom of browser clicking activates link attached div. anyway around this? thanks. <div class="directory_search_results"> <img src="images/no_photo.png" /> <ul class="staff_details"> <li class="search_results_name"><a href="http://www.netflix.com">micheal staff</a></li> <li class="search_results_location">university of illinois</li> <li class="search_results_email"><a href="mailto:test@test.org">test@test.com</a></li> <li class="search_results_phone">(407) 555-1212</li> <li class="search_results_office">(407) 555-1212</li> </ul></div> and jquery $(document).ready(function(){ $(".directory_search_results").click(function(){ ...
Read more

WPF: binding viewmodel property of type DateTime to Calendar inside ItemsControl -

Image
i have problem wpf binding. want bind list of months itemscontrol shows calendar control each month. each rendered calendar shows datetime.now,not bound datetimes. know why happening? this have far: the mainwindow.xaml <window x:class="calendarlisttest.mainwindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" title="mainwindow" height="350" width="525"> <grid> <itemscontrol x:name="calendarlist"> <itemscontrol.itemtemplate> <datatemplate> <calendar displaydate="{binding currentdate}" /> </datatemplate> </itemscontrol.itemtemplate> </itemscontrol> </grid> ** place collection assigned itemssource** private void window_loaded( object sender, routedeventargs e ) { cale...
Read more

java - Getting corefrences with Standard corenlp package -

i'm trying coreferences in text. i'm new corenlp package. tried code below, doesn't work, i'm open other methods well. /* * change template, choose tools | templates * , open template in editor. */ package corenlp; import edu.stanford.nlp.ling.coreannotations.collapsedccprocesseddependenciesannotation; import edu.stanford.nlp.ling.coreannotations.corefgraphannotation; import edu.stanford.nlp.ling.coreannotations.namedentitytagannotation; import edu.stanford.nlp.ling.coreannotations.partofspeechannotation; import edu.stanford.nlp.ling.coreannotations.sentencesannotation; import edu.stanford.nlp.ling.coreannotations.textannotation; import edu.stanford.nlp.ling.coreannotations.tokensannotation; import edu.stanford.nlp.ling.coreannotations.treeannotation; import edu.stanford.nlp.ling.corelabel; import edu.stanford.nlp.pipeline.*; import edu.stanford.nlp.trees.tree; import edu.stanford.nlp.trees.semgraph.semanticgraph; import edu.stanford.nlp.util.coremap; import e...
Read more