logoalt Hacker News

nasretdinov • yesterday at 4:33 PM • 1 reply • view on HN

Using a "work queue", i.e. a channel would still have a for loop like

  for filename := range workQueue {
      fp, err := os.Open(filename)
      if err != nil { ... }
      defer fp.Close()
      // do work
  }
Which would have the same exact problem :)

Replies

win311fwg • yesterday at 4:40 PM

I don't see the problem.

    for _, filename := range files {
        queue <- func() {
            f, _ := os.Open(filename)
            defer f.Close()
        }
    }
or more realistically,

    var group errgroup.Group
    group.SetLimit(10)
    for _, filename := range files {
        group.Go(func() error {
            f, err := os.Open(filename)
            if err != nil {
                return fmt.Errorf("failed to open file %s: %w", filename, err)
            }
            defer f.Close()  
            // ...
            return nil          
        })
    }
    if err := group.Wait(); err != nil {
        return fmt.Errorf("failed to process files: %w", err)
    }
Perhaps you can elaborate?

I did read your code, but it is not clear where the worker queue is. It looks like it ranges over (presumably) a channel of filenames, which is not meaningfully different than ranging over a slice of filenames. That is the original, non-concurrent solution, more or less.

➕ show 1 reply