Working with the File System in Deno

In this article, we’ll build on our introduction to Deno by creating a command-line tool that can search for text within files and folders. We’ll use a range of API methods that Deno provides to read and write to the file system.

In our last installment, we used Deno to build a command-line tool to make requests to a third-party API. In this article, we’re going to leave the network to one side and build a tool that lets you search the file system for text in files and folders within your current directory — similar to tools like grep.

Note: we’re not building a tool that will be as optimized and efficient as grep, nor are we aiming to replace it! The aim of building a tool like this is to get familiar with Deno’s file system APIs.

Installing Deno

We’re going to assume that you’ve got Deno up and running on your machine locally. You can check the Deno website or the previous article for more detailed installation instructions and also to get information on how to add Deno support to your editor of choice.

At the time of writing, the latest stable version of Deno is 1.10.2, so that’s what I’m using in this article.

For reference, you can find the complete code from this article on GitHub.

Setting Up Our New Command with Yargs

As in the previous article, we’ll use Yargs to build the interface that our users can use to execute our tool. Let’s create index.ts and populate it with the following:

import yargs from "https://deno.land/x/yargs@v17.0.1-deno/deno.ts";

interface Yargs<ArgvReturnType> {
  describe: (param: string, description: string) => Yargs<ArgvReturnType>;
  demandOption: (required: string[]) => Yargs<ArgvReturnType>;
  argv: ArgvReturnType;
}

interface UserArguments {
  text: string;
}

const userArguments: UserArguments =
  (yargs(Deno.args) as unknown as Yargs<UserArguments>)
    .describe("text", "the text to search for within the current directory")
    .demandOption(["text"])
    .argv;

console.log(userArguments);

There’s a fair bit going on here that’s worth pointing out:

• We install Yargs by pointing to its path on the Deno repository. I explicitly use a precise version number to make sure we always get that version, so that we don’t end up using whatever happens to be the latest version when the script runs.
• At the time of writing, the Deno + TypeScript experience for Yargs isn’t great, so I’ve created my own interface and used that to provide some type safety.
• UserArguments contains all the inputs we’ll ask the user for. For now, we’re only going to ask for text, but in future we could expand this to provide a list of files to search for, rather than assuming the current directory.

We can run this with deno run index.ts and see our Yargs output:

$ deno run index.ts
Check file:///home/jack/git/deno-file-search/index.ts
Options:
  --help     Show help                                                 [boolean]
  --version  Show version number                                       [boolean]
  --text     the text to search for within the current directory      [required]

Missing required argument: text

Now it’s time to get implementing!

Listing Files

Before we can start searching for text in a given file, we need to generate a list of directories and files to search within. Deno provides Deno.readdir, which is part of the “built-ins” library, meaning you don’t have to import it. It’s available for you on the global namespace.

Deno.readdir is asynchronous and returns a list of files and folders in the current directory. It returns these items as an AsyncIterator, which means we have to use the for await ... of loop to get at the results:

for await (const fileOrFolder of Deno.readDir(Deno.cwd())) {
  console.log(fileOrFolder);
}

This code will read from the current working directory (which Deno.cwd() gives us) and log each result. However, if you try to run the script now, you’ll get an error:

$ deno run index.ts --text='foo'
error: Uncaught PermissionDenied: Requires read access to <CWD>, run again with the --allow-read flag
for await (const fileOrFolder of Deno.readDir(Deno.cwd())) {
                                                   ^
    at deno:core/core.js:86:46
    at unwrapOpResult (deno:core/core.js:106:13)
    at Object.opSync (deno:core/core.js:120:12)
    at Object.cwd (deno:runtime/js/30_fs.js:57:17)
    at file:///home/jack/git/deno-file-search/index.ts:19:52

Remember that Deno requires all scripts to be explicitly given permissions to read from the file system. In our case, the --allow-read flag will enable our code to run:

~/$ deno run --allow-read index.ts --text='foo'
{ name: ".git", isFile: false, isDirectory: true, isSymlink: false }
{ name: ".vscode", isFile: false, isDirectory: true, isSymlink: false }
{ name: "index.ts", isFile: true, isDirectory: false, isSymlink: false }

In this case, I’m running the script in the directory where I’m building our tool, so it finds the TS source code, the .git repository and the .vscode folder. Let’s start writing some functions to recursively navigate this structure, as we need to find all the files within the directory, not just the top level ones. Additionally, we can add some common ignores. I don’t think anyone will want the script to search the entire .git folder!

In the code below, we’ve created the getFilesList function, which takes a directory and returns all files in that directory. If it encounters a directory, it will recursively call itself to find any nested files, and return the result:

const IGNORED_DIRECTORIES = new Set([".git"]);

async function getFilesList(
  directory: string,
): Promise<string[]> {
  const foundFiles: string[] = [];
  for await (const fileOrFolder of Deno.readDir(directory)) {
    if (fileOrFolder.isDirectory) {
      if (IGNORED_DIRECTORIES.has(fileOrFolder.name)) {
        // Skip this folder, it's in the ignore list.
        continue;
      }
      // If it's not ignored, recurse and search this folder for files.
      const nestedFiles = await getFilesList(
        `${directory}/${fileOrFolder.name}`,
      );
      foundFiles.push(...nestedFiles);
    } else {
      // We found a file, so store it.
      foundFiles.push(`${directory}/${fileOrFolder.name}`);
    }
  }
  return foundFiles;
}

We can then use this like so:

const files = await getFilesList(Deno.cwd());
console.log(files);

We also get some output that looks good:

$ deno run --allow-read index.ts --text='foo'
[
  "/home/jack/git/deno-file-search/.vscode/settings.json",
  "/home/jack/git/deno-file-search/index.ts"
]

Using the path Module

We’re could now combine file paths with template strings like so:

`${directory}/${fileOrFolder.name}`,

But it would be nicer to do this using Deno’s path module. This module is one of the modules that Deno provides as part of its standard library (much like Node does with its path module), and if you’ve used Node’s path module the code will look very similar. At the time of writing, the latest version of the std library Deno provides is 0.97.0, and we import the path module from the mod.ts file:

import * as path from "https://deno.land/std@0.97.0/path/mod.ts";

mod.ts is always the entrypoint when importing Deno’s standard modules. The documentation for this module lives on the Deno site and lists path.join, which will take multiple paths and join them into one path. Let’s import and use that function rather than manually combining them:

// import added to the top of our script
import yargs from "https://deno.land/x/yargs@v17.0.1-deno/deno.ts";
import * as path from "https://deno.land/std@0.97.0/path/mod.ts";

// update our usages of the function:
async function getFilesList(
  directory: string,
): Promise<string[]> {
  const foundFiles: string[] = [];
  for await (const fileOrFolder of Deno.readDir(directory)) {
    if (fileOrFolder.isDirectory) {
      if (IGNORED_DIRECTORIES.has(fileOrFolder.name)) {
        // Skip this folder, it's in the ignore list.
        continue;
      }
      // If it's not ignored, recurse and search this folder for files.
      const nestedFiles = await getFilesList(
        path.join(directory, fileOrFolder.name),
      );
      foundFiles.push(...nestedFiles);
    } else {
      // We found a file, so store it.
      foundFiles.push(path.join(directory, fileOrFolder.name));
    }
  }
  return foundFiles;
}

When using the standard library, it’s vital that you remember to pin to a specific version. Without doing so, your code will always load the latest version, even if that contains changes that will break your code. The Deno docs on the standard library go into this further, and I recommend giving that page a read.

Reading the Contents of a File

Unlike Node, which lets you read contents of files via the fs module and the readFile method, Deno provides readTextFile out of the box as part of its core, meaning that in this case we don’t need to import any additional modules. readTextFile does assume that the file is encoded as UTF-8 — which, for text files, is normally what you want. If you’re working with a different file encoding, you can use the more generic readFile, which doesn’t assume anything about the encoding and lets you pass in a specific decoder.

Once we’ve got the list of files, we can loop over them and read their contents as text:

const files = await getFilesList(Deno.cwd());

files.forEach(async (file) => {
  const contents = await Deno.readTextFile(file);
  console.log(contents);
});

Because we want to know the line number when we find a match, we can split the contents on a new line character (\n) and search each line in turn to see if there’s a match. That way, if there is, we’ll know the index of the line number so we can report it back to the user:

files.forEach(async (file) => {
  const contents = await Deno.readTextFile(file);
  const lines = contents.split("\n");
  lines.forEach((line, index) => {
    if (line.includes(userArguments.text)) {
      console.log("MATCH", line);
    }
  });
});

To store our matches, we can create an interface that represents a Match, and push matches onto an array when we find them:

interface Match {
  file: string;
  line: number;
}
const matches: Match[] = [];
files.forEach(async (file) => {
  const contents = await Deno.readTextFile(file);
  const lines = contents.split("\n");
  lines.forEach((line, index) => {
    if (line.includes(userArguments.text)) {
      matches.push({
        file,
        line: index + 1,
      });
    }
  });
});

Then we can log out the matches:

matches.forEach((match) => {
  console.log(match.file, "line:", match.line);
});

However, if you run the script now, and provide it with some text that will definitely match, you’ll still see no matches logged to the console. This is a common mistake people make with async and await within a forEach call; the forEach won’t wait for the callback to be complete before considering itself done. Take this code:

files.forEach(file => {
  new Promise(resolve => {
    ...
  })
})

The JavaScript engine is going to execute the forEach that runs on each file — generating a new promise — and then continue executing the rest of the code. It’s not going to automatically wait for those promises to resolve, and it’s exactly the same when we use await.

The good news is that this will work as expected in a for ... of loop, so rather than:

files.forEach(file => {...})

We can swap to:

for (const file of files) {
  ...
}

The for ... of loop will execute the code for each file in series, and upon seeing use of the await keyword it will pause execution until that promise has resolved. This means that after, the loop is executed, we know that all the promises have resolved, and now we do get matches logged onto the screen:

$ deno run --allow-read index.ts --text='readTextFile'
Check file:///home/jack/git/deno-file-search/index.ts
/home/jack/git/deno-file-search/index.ts line: 54

Let’s make some improvements to our output to make it easier to read. Rather than store matches as an array, let’s make it a Map where the keys are the filenames and the value is a Set of all the matches. That way, we can clarify our output by listing matches grouped by file, and have a data structure that lets us explore the data more easily.

First, we can create the data structure:

const matches = new Map<string, Set<Match>>();

Then we can store matches by adding them to a Set for that given file. This is a bit more work than before. We can’t just push items onto an array now. We firstly have to find any existing matches (or create a new Set) and then store them:

for (const file of files) {
  const contents = await Deno.readTextFile(file);
  const lines = contents.split("\n");
  lines.forEach((line, index) => {
    if (line.includes(userArguments.text)) {
      const matchesForFile = matches.get(file) || new Set<Match>();
      matchesForFile.add({
        file,
        line: index + 1,
      });
      matches.set(file, matchesForFile);
    }
  });
}

Then we can log the matches by iterating over the Map. When you use for ... of on a Map, each iteration gives you an array of two items, where the first is the key in the map and the second is the value:

for (const match of matches) {
  const fileName = match[0];
  const fileMatches = match[1];
  console.log(fileName);
  fileMatches.forEach((m) => {
    console.log("=>", m.line);
  });
}

We can do some destructuring to make this a little neater:

for (const match of matches) {
  const [fileName, fileMatches] = match;

Or even:

for (const [fileName, fileMatches] of matches) {

Now when we run the script we can see all the matches in a given file:

$ deno run --allow-read index.ts --text='Deno'
/home/jack/git/deno-file-search/index.ts
=> 15
=> 26
=> 45
=> 54

Finally, to make the output a bit clearer, let’s store the actual line that matched too. First, I’ll update my Match interface:

interface Match {
  file: string;
  lineNumber: number;
  lineText: string;
}

Then update the code that stores the matches. One really nice thing about TypeScript here is that you can update the Match interface and then have the compiler tell you the code you need to update. I’ll often update a type, and then wait for VS Code to highlight any problems. It’s a really productive way to work if you can’t quite remember all the places where the code needs an update:

if (line.includes(userArguments.text)) {
  const matchesForFile = matches.get(file) || new Set<Match>();
  matchesForFile.add({
    file,
    lineNumber: index + 1,
    lineText: line,
  });
  matches.set(file, matchesForFile);
}

The code that outputs the matches also needs an update:

for (const [fileName, fileMatches] of matches) {
  console.log(fileName);
  fileMatches.forEach((m) => {
    console.log("=>", m.lineNumber, m.lineText.trim());
  });
}

I decided to call trim() on our lineText so that, if the matched line is heavily indented, we don’t show it like that in the results. We’ll strip any leading (and trailing) whitespace in our output.

And with that, I’d say our first version is done!

$ deno run --allow-read index.ts --text='Deno'
Check file:///home/jack/git/deno-file-search/index.ts
/home/jack/git/deno-file-search/index.ts
=> 15 (yargs(Deno.args) as unknown as Yargs<UserArguments>)
=> 26 for await (const fileOrFolder of Deno.readDir(directory)) {
=> 45 const files = await getFilesList(Deno.cwd());
=> 55 const contents = await Deno.readTextFile(file);

Filtering by File Extension

Let’s extend the functionality so that users can filter the file extensions we match via an extension flag, which the user can pass an extension to (such as --extension js to only match .js files). First let’s update the Yargs code and the types to tell the compiler that we’re accepting an (optional) extension flag:

interface UserArguments {
  text: string;
  extension?: string;
}

const userArguments: UserArguments =
  (yargs(Deno.args) as unknown as Yargs<UserArguments>)
    .describe("text", "the text to search for within the current directory")
    .describe("extension", "a file extension to match against")
    .demandOption(["text"])
    .argv;

We can then update getFilesList so that it takes an optional second argument, which can be an object of configuration properties we can pass into the function. I often like functions to take an object of configuration items, as adding more items to that object is much easier than updating the function to require more parameters are passed in:

interface FilterOptions {
  extension?: string;
}

async function getFilesList(
  directory: string,
  options: FilterOptions = {},
): Promise<string[]> {}

Now in the body of the function, once we’ve found a file, we now check that either:

• The user didn’t provide an extension to filter by.
• The user did provide an extension to filter by, and the extension of the file matches what they provided. We can use path.extname, which returns the file extension for a given path (for foo.ts, it will return .ts, so we take the extension the user passed in and prepend a . to it).

async function getFilesList(
  directory: string,
  options: FilterOptions = {},
): Promise<string[]> {
  const foundFiles: string[] = [];
  for await (const fileOrFolder of Deno.readDir(directory)) {
    if (fileOrFolder.isDirectory) {
      if (IGNORED_DIRECTORIES.has(fileOrFolder.name)) {
        // Skip this folder, it's in the ignore list.
        continue;
      }
      // If it's not ignored, recurse and search this folder for files.
      const nestedFiles = await getFilesList(
        path.join(directory, fileOrFolder.name),
        options,
      );
      foundFiles.push(...nestedFiles);
    } else {
      // We know it's a file, and not a folder.

      // True if we weren't given an extension to filter, or if we were and the file's extension matches the provided filter.
      const shouldStoreFile = !options.extension ||
        path.extname(fileOrFolder.name) === `.${options.extension}`;

      if (shouldStoreFile) {
        foundFiles.push(path.join(directory, fileOrFolder.name));
      }
    }
  }
  return foundFiles;
}

Finally, we need to update our call to the getFilesList function, to pass it any parameters the user entered:

const files = await getFilesList(Deno.cwd(), userArguments);

Find and Replace

To finish off, let’s extend our tool to allow for basic replacement. If the user passes --replace=foo, we’ll take any matches we found from their search, and replace them with the provided word — in this case, foo, before writing that file to disk. We can use Deno.writeTextFile to do this. (Just like with readTextFile, you can also use writeFile if you need more control over the encoding.)

Once again, we’ll first update our Yargs code to allow the argument to be provided:

interface UserArguments {
  text: string;
  extension?: string;
  replace?: string;
}

const userArguments: UserArguments =
  (yargs(Deno.args) as unknown as Yargs<UserArguments>)
    .describe("text", "the text to search for within the current directory")
    .describe("extension", "a file extension to match against")
    .describe("replace", "the text to replace any matches with")
    .demandOption(["text"])
    .argv;

What we can now do is update our code that loops over each individual file to search for any matches. Once we’ve checked each line for a match, we can then use the replaceAll method (this is a relatively new method built into JavaScript) to take the contents of the file and swap each match out for the replacement text provided by the user:

for (const file of files) {
  const contents = await Deno.readTextFile(file);
  const lines = contents.split("\n");
  lines.forEach((line, index) => {
    if (line.includes(userArguments.text)) {
      const matchesForFile = matches.get(file) || new Set<Match>();
      matchesForFile.add({
        file,
        lineNumber: index + 1,
        lineText: line,
      });
      matches.set(file, matchesForFile);
    }
  });

  if (userArguments.replace) {
    const newContents = contents.replaceAll(
      userArguments.text,
      userArguments.replace,
    );
    // TODO: write to disk
  }
}

Writing to disk is a case of calling writeTextFile, providing the file path and the new contents:

if (userArguments.replace) {
  const newContents = contents.replaceAll(
    userArguments.text,
    userArguments.replace,
  );
  await Deno.writeTextFile(file, newContents);
}

When running this, however, we’ll now get a permissions error. Deno splits file reading and file writing into separate permissions, so you’ll need to pass the --allow-write flag to avoid an error:

$ deno run --allow-read index.ts --text='readTextFile' --extension=ts --replace='jackWasHere'
Check file:///home/jack/git/deno-file-search/index.ts
error: Uncaught (in promise) PermissionDenied: Requires write access to "/home/jack/git/deno-file-search/index.ts", run again with the --allow-write flag
    await Deno.writeTextFile(file, newContents);

You can pass --allow-write or be a little more specific with --allow-write=., which means the tool only has permission to write files within the current directory:

$ deno run --allow-read --allow-write=. index.ts --text='readTextFile' --extension=ts --replace='jackWasHere'
/home/jack/git/deno-file-search/index.ts
=> 74 const contents = await Deno.readTextFile(file);

Compiling to an Executable

Now that we have our script and we’re ready to share it, let’s ask Deno to bundle our tool into a single executable. This way, our end users won’t have to have Deno running and won’t have to pass in all the relevant permission flags every time; we can do that when bundling. deno compile lets us do this:

$ deno compile --allow-read --allow-write=. index.ts
Check file:///home/jack/git/deno-file-search/index.ts
Bundle file:///home/jack/git/deno-file-search/index.ts
Compile file:///home/jack/git/deno-file-search/index.ts
Emit deno-file-search

And then we can call the executable:

$ ./deno-file-search index.ts --text=readTextFile --extension=ts
/home/jack/git/deno-file-search/index.ts
=> 74 const contents = await Deno.readTextFile(file);

I really like this approach. We’re able to bundle the tool so our users don’t have to compile anything, and by providing the permissions up front we mean that users don’t have to. Of course, this is a trade-off. Some users might want to provide permissions such that they have full knowledge of what our script can and can’t do, but I think more often than not it’s good to feed the permissions into the executable.

Conclusion

I really do have a lot of fun working in Deno. Compared to Node, I love the fact that TypeScript, Deno Format, and other tools just come out the box. I don’t have to set up my Node project, then Prettier, and then figure out the best way to add TypeScript into that.

Deno is (unsurprisingly) not as polished or fleshed out as Node. Many third-party packages that exist in Node don’t have a good Deno equivalent (although I expect this will change in time), and at times the docs, whilst thorough, can be quite hard to find. But these are all small problems that you’d expect of any relatively new programming environment and language. I highly recommend exploring Deno and giving it a go. It’s definitely here to stay.

SitePoint has a growing list of articles on Deno. Check them out here if you’d like to explore Deno further.

FAQs About Working With the File System in Deno