Connect your shared folder

Principles in play: Track records, Control transitions.TRACETrack records · Control transitions

Every project has one shared place for large and bulky files: PDFs, datasets, recordings and exports. You say where it is, once, and your agent does the rest.

From 0.1.0-rc21

The shared folder arrived in TRACE-OS 0.1.0-rc21. A project set up with an earlier release gets it when it is updated.

Two routes

	A sync app folder	A bucket
What it is	A folder that OneDrive, Google Drive, Dropbox, SURFdrive or SURF Research Drive keeps in step on every coauthor’s computer.	Storage in Google Cloud Storage, SURF’s Object Store or another S3-compatible store. No app keeps a copy on your computer; your agent copies files in and out.
Who it suits	Most teams. Start here.	Data too large for a sync app, or a team that already has a bucket. It is the whole project’s choice: once the bucket is recorded, every coauthor uses it.
Cost	Usually none: your university or Google account already has it.	Paid. Google Cloud bills storage by the month and downloads by the gigabyte; SURF’s Object Store goes through your institution.
What you do once	Make a folder in the app and share it with your coauthors there.	Create the bucket and give coauthors access in the provider’s console (or ask IT), then sign in once when your agent sets up rclone.

SURFdrive and SURF Research Drive are synced folders, like OneDrive; only SURF’s Object Store is a bucket.

Steps

Sync app folder

1. Make a folder for the project in OneDrive or Google Drive, and share it with your coauthors there.

   Only you can do this step, because only you can sign in to your sync app.

2. Tell your agent where it is.

   Say to your agent

   Our shared folder is in OneDrive, called Patent study. Connect it.

   Not sure where it is on your computer? Say so: the agent lists the sync apps’ folders it can find and connects the one you confirm. It tells you in plain words what it will change before it changes anything.

3. Keep the folder on your computer.

   OneDrive and Google Drive keep files online until you open them. Right-click the project’s folder and choose Always keep on this device (OneDrive) or Available offline (Google Drive), so an analysis does not wait for downloads.

4. Each coauthor connects their own copy.

   When a coauthor’s work first needs the shared files, their agent checks whether the folder is connected on their computer, and if it is not, asks them where it is.

Bucket

1. Create the bucket and give your coauthors access.

   In the Google Cloud console, or through your institution for SURF’s Object Store. Only you or your IT department can do this step.

2. Tell your agent which bucket.

   Say to your agent

   Our shared files are in the Google Cloud bucket patent-study-data, folder raw. Set it up.

   The agent installs nothing without asking. It helps you install rclone, the tool that copies files to and from buckets, and walks you through signing in once. Your password or key stays in rclone’s own settings, never in the project.

3. Let the agent fetch and send files.

   Say to your agent

   Fetch the latest shared files.

   The agent shows what it will copy before it copies, because downloads from a bucket are billed. Nothing is deleted on either side, and a newer file is never overwritten.

4. Each coauthor sets up their own computer the same way.

   The bucket’s name is recorded in the project, so their agent knows where to look.

Everyone’s copy, checked against the record

A sync app can be halfway through a download, and two people can save the same file at once. So Git keeps a list of what the shared folder should hold: each file’s name, size and a fingerprint of its contents. When you add or change shared files, the agent updates that list and commits it with your work. When you pull a coauthor’s work, the agent checks your copy against the list and tells you what it found:

Say to your agent

Is my shared folder up to date?

The answer names the files still missing on your computer (usually still syncing), the ones that differ from the recorded version, and new ones nobody has recorded yet. If part of the folder cannot be read, the agent says so, and leaves the recorded list as it was rather than recording a shorter one. The agent also points out sync conflict copies, such as data-DESKTOP-1AB2C3D.csv next to data.csv, and asks you which version to keep; it deletes nothing itself.

Where things go

The agent decides, and says where it put each file.

Where	What goes there
Git	What the project writes and reviews: notes, code, coding records, trace records, the manuscript and slides.
The shared folder	Large or bulky material the team needs: PDFs, datasets, exports, recordings and transcripts the team may see.
The shared folder’s incoming/	Material from outside waiting for a decision: a delivered dataset, a provider’s corrected version, a reviewer’s files. It is not the project’s data until someone admits it.
Kept on your computer only	Anything else you want out of Git. Say “Keep the exports folder out of Git.” A file Git already tracks stays tracked: the agent names it, and whether to stop tracking it is your decision.
Neither	Passwords, keys and raw private transcripts.

Google Docs, Sheets and Slides sit in a Google Drive folder as links, not files. When an analysis needs one, the agent asks you to export it, for example as .csv.

When the results will be reported

The shared folder changes whenever a coauthor’s sync app copies something in. So before an analysis whose results will be reported, the agent takes a snapshot: a copy of the inputs, checked file by file, kept on your computer outside the shared folder, so that syncing does not change it. The agent then records the snapshot’s identifier as an input to the work, so a coauthor can find the same files later.

Say to your agent

Run the yearly shares again on a snapshot of the survey data, and record which snapshot you used.

If something goes wrong

What you see	What to say
The agent says the folder is not connected	Tell it where the folder is, as in step 2.
The agent says files are missing on this computer	Wait for your sync app to finish, then “Check my shared folder again.” For a bucket, “Fetch the latest shared files.”
The agent says a file differs from the recorded version	Ask “Who changed it, and which version should we use?”
The agent found conflict copies	Decide which version to keep; the agent can tidy up once you say so.
The agent says the project itself is in a synced folder	Ask it to help you move the project out first. Git and sync apps both change files, and they get in each other’s way.
You use Linux	OneDrive and Google Drive have no Linux app. Ask the agent to make the team’s folder a local folder with rclone and connect that, so your coauthors keep their sync apps.

Things to know

• Your path stays yours. Each person’s folder sits in a different place on their computer, so the link stays out of Git.
• Ignored is not private. Who can see the files is decided in the sync app or the bucket’s permissions, not by Git.
• Admitting is still recorded. A file that enters the shared folder from outside is still an input: it waits in incoming/ until someone admits, returns or refuses it; see Bring in outside material.

What the agent runs

python scripts/shared_folder.py status                     # where the shared files are, the next step, and what waits in incoming/
python scripts/shared_folder.py find                       # the sync apps' folders on this computer
python scripts/shared_folder.py connect "<folder>" --plan  # then without --plan; --move-existing, --create, --replace
python scripts/shared_folder.py bucket set <remote>:<bucket>/<folder>   # a bucket through rclone instead
python scripts/shared_folder.py pull --plan                # then pull; push likewise; never deletes
python scripts/shared_folder.py check                      # conflict copies, online-only files, shortcuts
python scripts/shared_folder.py inventory                  # sources/shared-inventory.json, for Git
python scripts/shared_folder.py verify                     # this computer's copy against that inventory
python scripts/shared_folder.py snapshot <subfolder> --dataset <name>   # a checked local copy under workspace/snapshots/
python scripts/shared_folder.py keep-local "<pattern>"     # keep a path out of Git
python scripts/shared_folder.py disconnect                 # remove the link; the folder is untouched

The workflow check reports where the shared files are on this computer as information; it never fails on it.