Skip to content

Listing and searching

storix has one directory listing for every shape you need: eager or lazy, names or rich entries, one level or recursive. They all share the same semantics (hidden entries hidden unless all=True, a session-relative path), so the only choice is how much you want and how you want it.

Pick one

Method Depth Yields Eager/lazy
ls one directory list[StorixPath] (names) eager
iterdir one directory StorixPath (names) lazy
scandir one directory DirEntry (name, path, kind, size) lazy
walk recursive DirEntry lazy
find recursive DirEntry, filtered lazy
glob recursive StorixPath, pattern-matched lazy

The rule of thumb:

  • Just the names of one directory? ls (you get a list) or iterdir (you get a generator, for a huge directory you do not want to materialize).
  • Names and kind/size of one directory? scandir. It carries the kind and any size the backend listed for free, so you never stat an entry just to tell a file from a directory.
  • Everything underneath? walk (all of it), find (filtered), or glob (a path pattern). All stream, so a million-entry tree flows through bounded memory.
from storix import get_storage

fs = get_storage("local")

# one directory
for name in fs.ls("/data"): ...                 # eager names
async for path in fs.iterdir("/data"): ...      # (aio) lazy names
for entry in fs.scandir("/data"):               # lazy, rich
    print(entry.name, entry.kind, entry.size)

# recursive
for entry in fs.walk("/data"): ...              # every descendant
for entry in fs.find("/data", name="*.py", kind="file"): ...
for path in fs.glob("**/*.parquet", "/data"): ...

find vs glob

They are the same recursive walk with two matching styles:

  • find(name=..., kind=...) matches the basename (fnmatch, so '*.py', 'config.*') and can restrict to files or directories. It yields rich DirEntry objects.
  • glob(pattern) matches a path pattern (*, ?, **), pathlib-style, and yields paths. '*.py' is direct children, '**/*.py' is every depth.

Reach for find when you want the kind/size of each hit or a type filter; glob when you are thinking in path patterns.

Both exclude hidden entries (and anything under a hidden directory) by default, like the rest of the family - pass all=True to reach dotfiles: fs.find(name=".env", all=True).

DirEntry is not a stat

scandir/walk/find give you DirEntry (name, path, kind, size) - what a listing produces for free. For the full picture (timestamps, custom metadata) call fs.stat(entry.path), which returns FileProperties. Keeping them separate is deliberate: a listing of a thousand entries costs one request; a thousand stats costs a thousand.

A note on cost

ls and its siblings are one listing request per directory. On local disk that is a cheap syscall; on an object store it is a network round trip. So:

  • A recursive call (walk/find/glob/tree) lists every directory it visits - one round trip each on cloud. Filter early (find(name=...) stops yielding, but still walks) and prefer a bounded depth when you can.
  • The sx CLI's empty/full folder icon (dir_contents) uses empty_children. Object-store backends that advertise bulk listing derive every immediate child directory's state from one recursive listing. Other backends probe the children concurrently, and very large subtrees fall back to that portable path once the bulk key bound is exceeded. Turn the icon off with dir_contents = false (or --no-icons) when even that extra work is unwanted, and keep a cache layer active in interactive sessions so repeated metadata lookups are free.

See Cache with Redis or disk for the read-through cache, and The sx CLI for the shell.