Skip to content

Commit 171c6c7

Browse files
walterbm-coherewalterbm-cohere
committed
daulet -> cohere-ai
1 parent eafe2ce commit 171c6c7

13 files changed

Lines changed: 25 additions & 25 deletions

File tree

‎BUILD.bazel‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ load("@crate_index//:defs.bzl", "aliases", "all_crate_deps")
33
load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
44
load("@rules_rust//rust:defs.bzl", "rust_static_library")
55

6-
# gazelle:prefix github.com/daulet/tokenizers
6+
# gazelle:prefix github.com/cohere-ai/tokenizers
77
gazelle(
88
name = "gazelle",
99
)
@@ -53,6 +53,6 @@ go_library(
5353
":tokenizers_rs",
5454
],
5555
cgo = True,
56-
importpath = "github.com/daulet/tokenizers",
56+
importpath = "github.com/cohere-ai/tokenizers",
5757
visibility = ["//visibility:public"],
5858
)

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ It should look something like this:
1717
```
1818
goos: darwin
1919
goarch: arm64
20-
pkg: github.com/daulet/tokenizers
20+
pkg: github.com/cohere-ai/tokenizers
2121
│ benchmarks/786da4095f5ca3d598db1236c46401b63874f640.txt │ benchmarks/38a9a14c1c56b113461b0c7350c72de949e23cc2.txt │
2222
│ sec/op │ sec/op vs base │
2323
EncodeNTimes-10 13.26µ ± 4% 13.11µ ± 1% -1.09% (p=0.041 n=6)

‎README.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,11 @@ Go bindings for the [HuggingFace Tokenizers](https://github.com/huggingface/toke
88

99
### Using pre-built binaries
1010

11-
If you don't want to install Rust toolchain, build it in docker: `docker build --platform=linux/amd64 -f release/Dockerfile .` or use prebuilt binaries from the [releases](https://github.com/daulet/tokenizers/releases) page. Prebuilt libraries are available for:
11+
If you don't want to install Rust toolchain, build it in docker: `docker build --platform=linux/amd64 -f release/Dockerfile .` or use prebuilt binaries from the [releases](https://github.com/cohere-ai/tokenizers/releases) page. Prebuilt libraries are available for:
1212

13-
* [darwin-arm64](https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.darwin-arm64.tar.gz)
14-
* [linux-arm64](https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.linux-arm64.tar.gz)
15-
* [linux-amd64](https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.linux-amd64.tar.gz)
13+
* [darwin-arm64](https://github.com/cohere-ai/tokenizers/releases/latest/download/libtokenizers.darwin-arm64.tar.gz)
14+
* [linux-arm64](https://github.com/cohere-ai/tokenizers/releases/latest/download/libtokenizers.linux-arm64.tar.gz)
15+
* [linux-amd64](https://github.com/cohere-ai/tokenizers/releases/latest/download/libtokenizers.linux-amd64.tar.gz)
1616

1717
## Getting started
1818

@@ -21,7 +21,7 @@ TLDR: [working example](example/main.go).
2121
Load a tokenizer from a JSON config:
2222

2323
```go
24-
import "github.com/daulet/tokenizers"
24+
import "github.com/cohere-ai/tokenizers"
2525

2626
tk, err := tokenizers.FromFile("./data/bert-base-uncased.json")
2727
if err != nil {
@@ -55,13 +55,13 @@ go test . -bench=. -benchmem -benchtime=10s
5555

5656
goos: darwin
5757
goarch: arm64
58-
pkg: github.com/daulet/tokenizers
58+
pkg: github.com/cohere-ai/tokenizers
5959
BenchmarkEncodeNTimes-10 959494 12622 ns/op 232 B/op 12 allocs/op
6060
BenchmarkEncodeNChars-10 1000000000 2.046 ns/op 0 B/op 0 allocs/op
6161
BenchmarkDecodeNTimes-10 2758072 4345 ns/op 96 B/op 3 allocs/op
6262
BenchmarkDecodeNTokens-10 18689725 648.5 ns/op 7 B/op 0 allocs/op
6363
PASS
64-
ok github.com/daulet/tokenizers 126.681s
64+
ok github.com/cohere-ai/tokenizers 126.681s
6565
```
6666

6767
Run equivalent Rust tests with `cargo bench`.

‎example/BUILD.bazel‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
33
go_library(
44
name = "example_lib",
55
srcs = ["main.go"],
6-
importpath = "github.com/daulet/tokenizers/example",
6+
importpath = "github.com/cohere-ai/tokenizers/example",
77
visibility = ["//visibility:private"],
88
deps = ["//:tokenizers"],
99
)

‎example/Dockerfile‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@ FROM golang:1.21 as builder-go
44
ARG TARGETPLATFORM
55
ARG VERSION=v0.6.0
66
WORKDIR /workspace
7-
RUN curl -fsSL https://github.com/daulet/tokenizers/releases/download/${VERSION}/libtokenizers.$(echo ${TARGETPLATFORM} | tr / -).tar.gz | tar xvz
7+
RUN curl -fsSL https://github.com/cohere-ai/tokenizers/releases/download/${VERSION}/libtokenizers.$(echo ${TARGETPLATFORM} | tr / -).tar.gz | tar xvz
88
COPY ./example .
99
COPY ./test/data ./test/data
1010
RUN go mod download
11-
RUN mv ./libtokenizers.a /go/pkg/mod/github.com/daulet/tokenizers@${VERSION}/lib/$(echo ${TARGETPLATFORM} | tr / -)/libtokenizers.a
11+
RUN mv ./libtokenizers.a /go/pkg/mod/github.com/cohere-ai/tokenizers@${VERSION}/lib/$(echo ${TARGETPLATFORM} | tr / -)/libtokenizers.a
1212
# mounting Go cache won't work since we mutate it above
1313
RUN go run main.go

‎example/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
```
44
# Keep this version in sync with go module release
5-
curl -fsSL https://github.com/daulet/tokenizers/releases/download/v0.9.0/libtokenizers.darwin-aarch64.tar.gz | tar xvz
5+
curl -fsSL https://github.com/cohere-ai/tokenizers/releases/download/v0.9.0/libtokenizers.darwin-aarch64.tar.gz | tar xvz
66
# change -L argument to where you've placed the library download above
77
go run -ldflags="-extldflags '-L$(pwd)'" main.go
88
```

‎example/go.mod‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
module github.com/daulet/tokenizers/example
1+
module github.com/cohere-ai/tokenizers/example
22

33
go 1.22
44

5-
require github.com/daulet/tokenizers v0.9.0
5+
require github.com/cohere-ai/tokenizers v0.9.0

‎example/go.sum‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
github.com/daulet/tokenizers v0.9.0 h1:PSjFUGeuhqb3C0GKP9hdvtHvJ6L1AZceV+0nYGACtCk=
2-
github.com/daulet/tokenizers v0.9.0/go.mod h1:tGnMdZthXdcWY6DGD07IygpwJqiPvG85FQUnhs/wSCs=
1+
github.com/cohere-ai/tokenizers v0.9.0 h1:PSjFUGeuhqb3C0GKP9hdvtHvJ6L1AZceV+0nYGACtCk=
2+
github.com/cohere-ai/tokenizers v0.9.0/go.mod h1:tGnMdZthXdcWY6DGD07IygpwJqiPvG85FQUnhs/wSCs=
33
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
44
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
55
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=

‎example/main.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ package main
33
import (
44
"fmt"
55

6-
"github.com/daulet/tokenizers"
6+
"github.com/cohere-ai/tokenizers"
77
)
88

99
func main() {

‎go.mod‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
module github.com/daulet/tokenizers
1+
module github.com/cohere-ai/tokenizers
22

33
go 1.18
44

@@ -8,4 +8,4 @@ require (
88
github.com/davecgh/go-spew v1.1.1 // indirect
99
github.com/pmezard/go-difflib v1.0.0 // indirect
1010
gopkg.in/yaml.v3 v3.0.1 // indirect
11-
)
11+
)

0 commit comments

Comments
 (0)