[ArchEc] current work

[Thomas Krichel]

  Hi gang,

  I'm still working on the tarballs of ArchEc. Here is a
  submission to a meeting explains where I am heading to
  in non-technical terms. The technical details are still
  progress, even after two months. It's just that there is
  a lot of junk to fight with in the RePEc tarballs.
 
  -----------------------------------------------------------------
  
  In digital archiving, it is quite common to start with records in
  files. If each file contains just one record, then preserving the
  files is essentially the same as preserving the records. But what if
  the files contain several records? And what if the files change over
  time? For example, if the files are harvested from a bunch of
  sources. These sources maybe creating poorly-formed records. Each
  source offers files that contain records. We can harvest the current
  versions of these files and they land up on our disk ... how to
  preserve them?
  
  One approach is to preserve the files. In that case we loose no
  information. We can just look at what files are changing, and copy the
  changed ones into an archival location. I see two problems. (1) if
  there are files in which records are accumulating, we are wasting disk
  space because we store the same record over and over again. (2) what
  about consumers of our archival material? They presumably do not care
  about the files. They want the records. If we just tell them, “hey,
  here are the files, go figure”, we are not likely to get much buy-in.
  
  Another approach is to split the files into records, and preserve the
  records. Then we solve the two problems with the file-based
  approach. But we get more serious problems. We loose any information
  that was tied to the fact that records where in the same file. If the
  software used to split the records was buggy, or if we change our
  opinion about record borders, or the nature of records, we have no way
  to get back. We could circumvent that we some form of clever
  metadata. It has to be clever because the data may be broken in all
  sort of ways.
  
  The prior art that lead me to this problem is the ArchEc
  http://archec.repec.org project. It is a humble effort to preserve
  RePEc http://repec.org data. A first stage received €3000 funding from
  the Fondation Banque de France pour la recherche économique. In this
  stage, I worked on preserving full-text instances pointed to in the
  RePEc data. In the current stage, funded with €2000 by the same
  funder, the work ultimately aims to preserve the actual RePEc
  records. But even in the funding application, available at
  http://governance.repec.org/applications/lebach.docx, I only pledged
  to work on preserving files, because I was unsure about how to
  preserve records.
  
  Well, on 2 August 2021, I had an ingenious idea how to preserve files
  and records. I have implemented it. Thus, I have theory, software and
  a wealth of experience that I will share during the talk. While the
  work is obviously made for RePEc, the conceptual framework and the
  methods used apply to any time-varying collections of records that are
  in files. And the data moves into WARCs. I guess that the audience
  will be familiar with that format.

-- 

  Cheers,

  Thomas Krichel                  http://openlib.org/home/krichel
                                              skype:thomaskrichel