[ArchEc] Internet Archive and RePEc

[Thomas Krichel]

  Bryan Newbold writes

> RePEc has been on our list to harvest and crawl for some time,

  RePEc is not something you can crawl, in the web of the term.
  Your best bet is to download RePEc metadata and read that.

  As Christian Zimmermann has hinted, I am working on ArchEc a project
  to digitally archive RePEc. I intend to archive all of RePEc, not
  just working papers, but also the metadata records that we
  have. Progress has been extremly modest because of lack of resources. In
  2019, we got a 3k Euro subsidy. I am in the second year of working
  on this. The software I wrote in intended to be fairly generic, in
  the sense that it takes a pile of metadata, extracts full text
  links. My severe mistake was to use wget to do the actual downloads.
  A lot of current code is checks and corrects on wget output.  But I
  have a first set of WARC files available.  You can get them with

krichel at trabbi/tmp$ rsync -av --exclude '*.cdx' rsync://archec.repec.org/vault/ vault/

  The vault has the actual archives.  I will have roughly about 900k
  docs available by the end of the year.

  I also provide payload indexing data "plind", aggregated by RePEc
  series for user services to deliver full text out of the archives,
  reachable by

krichel at trabbi/tmp$ rsync -av  rsync://archec.repec.org/plind/ plind/

> Do you know anybody with familiarity of both RePEc and programming
> (eg, Python)

  Me. I code in Python, XSLT, Perl, Javascript... 

> who would be interested in taking a stab at this?

  Interested yes, but if there no funding for it, it will have to wait
  until the rest of ArchEc is done. That's a matter of years rather
  than months. That said, which a current income of about 600 Euros
  a month, I'm ready to work. 

-- 

  Cheers,

  Thomas Krichel                  http://openlib.org/home/krichel
                                              skype:thomaskrichel