Skip to content

tingweiwang / nfs-storageclass

Go to a project
Commit feac941e by tingweiwang
Options

nfs storage class

0 parents
Expand all
Inline Side-by-side
Showing with 2713 additions and 0 deletions
nfs-client/.gitignore 0 → 100644
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
*.prof
nfs-client-provisioner
nfs-client/CHANGELOG.md 0 → 100644
# v2.0.1
- Add support for ARM (Raspberry PI). Image at `quay.io/external_storage/nfs-client-provisioner-arm`. (https://github.com/kubernetes-incubator/external-storage/pull/275)
# v2.0.0
- Fix issue 149 - nfs-client-provisioner create folder with 755, not 777 (https://github.com/kubernetes-incubator/external-storage/pull/150)
# v1
- Initial release
\ No newline at end of file
nfs-client/Makefile 0 → 100644
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
ifeq ($(REGISTRY),)
REGISTRY = quay.io/external_storage/
endif
ifeq ($(VERSION),)
VERSION = latest
endif
IMAGE = $(REGISTRY)nfs-client-provisioner:$(VERSION)
IMAGE_ARM = $(REGISTRY)nfs-client-provisioner-arm:$(VERSION)
MUTABLE_IMAGE = $(REGISTRY)nfs-client-provisioner:latest
MUTABLE_IMAGE_ARM = $(REGISTRY)nfs-client-provisioner-arm:latest
all: build image build_arm image_arm
container: build image build_arm image_arm
build:
CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o docker/x86_64/nfs-client-provisioner ./cmd/nfs-client-provisioner
build_arm:
CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -a -ldflags '-extldflags "-static"' -o docker/arm/nfs-client-provisioner ./cmd/nfs-client-provisioner
image:
docker build -t $(MUTABLE_IMAGE) docker/x86_64
docker tag $(MUTABLE_IMAGE) $(IMAGE)
image_arm:
docker run --rm --privileged multiarch/qemu-user-static:register --reset
docker build -t $(MUTABLE_IMAGE_ARM) docker/arm
docker tag $(MUTABLE_IMAGE_ARM) $(IMAGE_ARM)
push:
docker push $(IMAGE)
docker push $(MUTABLE_IMAGE)
docker push $(IMAGE_ARM)
docker push $(MUTABLE_IMAGE_ARM)
nfs-client/OWNERS 0 → 100644
approvers:
- jackielii
nfs-client/README.md 0 → 100644
# Kubernetes NFS-Client Provisioner
[![Docker Repository on Quay](https://quay.io/repository/external_storage/nfs-client-provisioner/status "Docker Repository on Quay")](https://quay.io/repository/external_storage/nfs-client-provisioner)
**nfs-client** is an automatic provisioner that use your *existing and already configured* NFS server to support dynamic provisioning of Kubernetes Persistent Volumes via Persistent Volume Claims. Persistent volumes are provisioned as ``${namespace}-${pvcName}-${pvName}``.
# How to deploy nfs-client to your cluster.
To note again, you must *already* have an NFS Server.
## With Helm
Follow the instructions for the stable helm chart maintained at https://github.com/helm/charts/tree/master/stable/nfs-client-provisioner
The tl;dr is
```bash
$ helm install stable/nfs-client-provisioner --set nfs.server=x.x.x.x --set nfs.path=/exported/path
```
## Without Helm
**Step 1: Get connection information for your NFS server**. Make sure your NFS server is accessible from your Kubernetes cluster and get the information you need to connect to it. At a minimum you will need its hostname.
**Step 2: Get the NFS-Client Provisioner files**. To setup the provisioner you will download a set of YAML files, edit them to add your NFS server's connection information and then apply each with the ``kubectl`` / ``oc`` command.
Get all of the files in the [deploy](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client/deploy) directory of this repository. These instructions assume that you have cloned the [external-storage](https://github.com/kubernetes-incubator/external-storage) repository and have a bash-shell open in the ``nfs-client`` directory.
**Step 3: Setup authorization**. If your cluster has RBAC enabled or you are running OpenShift you must authorize the provisioner. If you are in a namespace/project other than "default" edit `deploy/rbac.yaml`.
Kubernetes:
```sh
# Set the subject of the RBAC objects to the current namespace where the provisioner is being deployed
$ NS=$(kubectl config get-contexts|grep -e "^\*" |awk '{print $5}')
$ NAMESPACE=${NS:-default}
$ sed -i'' "s/namespace:.*/namespace: $NAMESPACE/g" ./deploy/rbac.yaml
$ kubectl create -f deploy/rbac.yaml
```
OpenShift:
On some installations of OpenShift the default admin user does not have cluster-admin permissions. If these commands fail refer to the OpenShift documentation for **User and Role Management** or contact your OpenShift provider to help you grant the right permissions to your admin user.
```sh
# Set the subject of the RBAC objects to the current namespace where the provisioner is being deployed
$ NAMESPACE=`oc project -q`
$ sed -i'' "s/namespace:.*/namespace: $NAMESPACE/g" ./deploy/rbac.yaml
$ oc create -f deploy/rbac.yaml
$ oadm policy add-scc-to-user hostmount-anyuid system:serviceaccount:$NAMESPACE:nfs-client-provisioner
```
**Step 4: Configure the NFS-Client provisioner**
Note: To deploy to an ARM-based environment, use: `deploy/deployment-arm.yaml` instead, otherwise use `deploy/deployment.yaml`.
Next you must edit the provisioner's deployment file to add connection information for your NFS server. Edit `deploy/deployment.yaml` and replace the two occurences of <YOUR NFS SERVER HOSTNAME> with your server's hostname.
```yaml
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: <YOUR NFS SERVER HOSTNAME>
- name: NFS_PATH
value: /var/nfs
volumes:
- name: nfs-client-root
nfs:
server: <YOUR NFS SERVER HOSTNAME>
path: /var/nfs
```
You may also want to change the PROVISIONER_NAME above from ``fuseim.pri/ifs`` to something more descriptive like ``nfs-storage``, but if you do remember to also change the PROVISIONER_NAME in the storage class definition below:
This is `deploy/class.yaml` which defines the NFS-Client's Kubernetes Storage Class:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false" # When set to "false" your PVs will not be archived
# by the provisioner upon deletion of the PVC.
```
**Step 5: Finally, test your environment!**
Now we'll test your NFS provisioner.
Deploy:
```sh
$ kubectl create -f deploy/test-claim.yaml -f deploy/test-pod.yaml
```
Now check your NFS Server for the file `SUCCESS`.
```sh
kubectl delete -f deploy/test-pod.yaml -f deploy/test-claim.yaml
```
Now check the folder has been deleted.
**Step 6: Deploying your own PersistentVolumeClaims**. To deploy your own PVC, make sure that you have the correct `storage-class` as indicated by your `deploy/class.yaml` file.
For example:
```yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim
annotations:
volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Mi
```
nfs-client/deploy/class.yaml 0 → 100644
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false"
nfs-client/deploy/deployment-arm.yaml 0 → 100644
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner-arm:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 10.10.10.60
- name: NFS_PATH
value: /ifs/kubernetes
volumes:
- name: nfs-client-root
nfs:
server: 10.10.10.60
path: /ifs/kubernetes
nfs-client/deploy/deployment.yaml 0 → 100644
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 10.10.10.60
- name: NFS_PATH
value: /ifs/kubernetes
volumes:
- name: nfs-client-root
nfs:
server: 10.10.10.60
path: /ifs/kubernetes
nfs-client/deploy/objects/README.md 0 → 100644
The objects in this directory are the same as in the parent except split up into one file per object for certain users' convenience.
nfs-client/deploy/objects/class.yaml 0 → 100644
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false"
nfs-client/deploy/objects/clusterrole.yaml 0 → 100644
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
nfs-client/deploy/objects/clusterrolebinding.yaml 0 → 100644
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
nfs-client/deploy/objects/deployment-arm.yaml 0 → 100644
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner-arm:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 10.10.10.60
- name: NFS_PATH
value: /ifs/kubernetes
volumes:
- name: nfs-client-root
nfs:
server: 10.10.10.60
path: /ifs/kubernetes
nfs-client/deploy/objects/deployment.yaml 0 → 100644
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 10.10.10.60
- name: NFS_PATH
value: /ifs/kubernetes
volumes:
- name: nfs-client-root
nfs:
server: 10.10.10.60
path: /ifs/kubernetes
nfs-client/deploy/objects/role.yaml 0 → 100644
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
nfs-client/deploy/objects/rolebinding.yaml 0 → 100644
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
nfs-client/deploy/objects/serviceaccount.yaml 0 → 100644
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
nfs-client/deploy/rbac.yaml 0 → 100644
kind: ServiceAccount
apiVersion: v1
metadata:
name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
nfs-client/deploy/test-claim.yaml 0 → 100644
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim
annotations:
volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Mi
nfs-client/deploy/test-pod.yaml 0 → 100644
kind: Pod
apiVersion: v1
metadata:
name: test-pod
spec:
containers:
- name: test-pod
image: gcr.io/google_containers/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: nfs-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: test-claim
nfs-client/docker/arm/Dockerfile 0 → 100644
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM hypriot/rpi-alpine:3.6
RUN apk update --no-cache && apk add ca-certificates
COPY nfs-client-provisioner /nfs-client-provisioner
ENTRYPOINT ["/nfs-client-provisioner"]
nfs-client/docker/x86_64/Dockerfile 0 → 100644
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM alpine:3.6
RUN apk update --no-cache && apk add ca-certificates
COPY nfs-client-provisioner /nfs-client-provisioner
ENTRYPOINT ["/nfs-client-provisioner"]
nfs/CHANGELOG.md 0 → 100644
# v1.0.8
- Add mountOptions StorageClass parameter (#84) (see [Usage](./docs/usage.md) for complete SC parameter info)
- Replace root-squash argument with a rootSquash SC parameter (#86) (see [Usage](./docs/usage.md) for complete SC parameter info)
- If the root-squash argument is specified, the provisioner will fail to start; please if you're using it, convert to the SC parameter before updating!
- Watch for unexpected stop of ganesha.nfsd and restart if seen (#98). This is a simple health check that mitigates NFS ganesha crashes which are under investigation (but probably out of the provisioner's control to prevent at the moment).
# v1.0.7
- Set a high limit for maximum number of files Ganesha may have open (setrlimit RLIMIT_NOFILE) -- this requires the additional SYS_RESOURCE capability, if not available the provisioner will still start but with a warning
# v1.0.6
- Reduce image size by a lot
# v1.0.5
- Add compatibility with kubernetes v1.6.x (using lib v2.0.x)
# v1.0.4
- Add `server-hostname` flag
# Rename kubernetes-incubator/nfs-provisioner to kubernetes-incubator/external-storage
- The previous releases were done when the repo was named nfs-provisioner: http://github.com/kubernetes-incubator/nfs-provisioner/releases. Newer releases done here in external-storage will *not* have corresponding git tags (external-storage's git tags are reserved for versioning the library), so to keep track of releases check this changelog, the [README](README.md), or [Quay](https://quay.io/repository/kubernetes_incubator/nfs-provisioner)
# v1.0.3
- Fix inability to export NFS shares ("1 validation errors in block FSAL") when using Docker's overlay storage driver (CoreOS/container linux, GCE) by patching Ganesha to use device number as fsid. (#63)
- Adds configurable number of retries on failed Provisioner operations. Configurable as an argument to `NewProvisionController`. nfs-provisioner defaults to 10 retries unless the new flag/argument is used. (#65)
# v1.0.2
- Usage demo & how-to for writing your own external PV provisioner added here https://github.com/kubernetes-incubator/nfs-provisioner/tree/master/demo
- Change behaviour for getting NFS server IP from env vars (node, service) in case POD_IP env var is not set when needed. Use `hostname -i` as a fallback only for when running out-of-cluster (#52)
- Pass whole PVC object from controller to `Provision` as part of `VolumeOptions`, like upstream (#48)
- Filter out controller's self-generated race-to-lock leader election PVC updates from being seen as forced resync PVC updates (#58)
- Fix controller's event watching for ending race-to-lock leader elections early. Now correctly discover the latest `ProvisionFailed`/`ProvisionSucceeded` events on a claim (#59)
# v1.0.1
- Add rootsquash flag for enabling/disabling rootsquash https://github.com/kubernetes-incubator/nfs-provisioner/pull/40
# v1.0.0
- Automatically create NFS PVs for any user-defined Storage Class of PVCs, backed by a containerized NFS server that creates & exports shares from some user-defined mounted storage
- Support multiple ways to run:
- standalone Pod, e.g. for easy dynamically provisioned scratch space
- stateful app, either as a StatefulSet or Deployment of 1 replica: the NFS server will survive restarts and its provisioned PVs can be backed by some mounted persistent storage e.g. a hostPath or one big PV
- DaemonSet, where each node runs the NFS server to expose its hostPath storage
- Docker container or binary outside of Kube
- Race-to-lock PVCs: when multiple instances are running & serving the same PVCs, only one attempts to provision for a PVC at a time
- Optionally exponentially backoff from calls to Provision() and Delete()
- Optionally set per-PV filesystem quotas: based on XFS project-level quotas and available only when running outside of Kubernetes (pending mount option support in Kube)
Docker image:
`quay.io/kubernetes_incubator/nfs-provisioner:v1.0.0`
nfs/Makefile 0 → 100644
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
ifeq ($(REGISTRY),)
REGISTRY = quay.io/kubernetes_incubator/
endif
ifeq ($(VERSION),)
VERSION = latest
endif
IMAGE = $(REGISTRY)nfs-provisioner:$(VERSION)
MUTABLE_IMAGE = $(REGISTRY)nfs-provisioner:latest
all build:
GO111MODULE=on GOOS=linux go build ./cmd/nfs-provisioner
.PHONY: all build
container: build quick-container
.PHONY: container
quick-container:
cp nfs-provisioner deploy/docker/nfs-provisioner
docker build -t $(MUTABLE_IMAGE) deploy/docker
docker tag $(MUTABLE_IMAGE) $(IMAGE)
.PHONY: quick-container
push: container
docker push $(IMAGE)
docker push $(MUTABLE_IMAGE)
.PHONY: push
test-all: test test-e2e
test:
go test `go list ./... | grep -v 'vendor\|test\|demo'`
.PHONY: test
test-e2e:
cd ./test/e2e; ./test.sh
.PHONY: test-e2e
clean:
rm -f nfs-provisioner
rm -f deploy/docker/nfs-provisioner
rm -rf test/e2e/vendor
.PHONY: clean
nfs/OWNERS 0 → 100644
approvers:
- wongma7
- jsafrane
nfs/README.md 0 → 100644
# nfs-provisioner
[![Docker Repository on Quay](https://quay.io/repository/kubernetes_incubator/nfs-provisioner/status "Docker Repository on Quay")](https://quay.io/repository/kubernetes_incubator/nfs-provisioner)
```
quay.io/kubernetes_incubator/nfs-provisioner
```
nfs-provisioner is an out-of-tree dynamic provisioner for Kubernetes 1.4+. You can use it to quickly & easily deploy shared storage that works almost anywhere. Or it can help you write your own out-of-tree dynamic provisioner by serving as an example implementation of the requirements detailed in [the proposal](https://github.com/kubernetes/kubernetes/pull/30285).
It works just like in-tree dynamic provisioners: a `StorageClass` object can specify an instance of nfs-provisioner to be its `provisioner` like it specifies in-tree provisioners such as GCE or AWS. Then, the instance of nfs-provisioner will watch for `PersistentVolumeClaims` that ask for the `StorageClass` and automatically create NFS-backed `PersistentVolumes` for them. For more information on how dynamic provisioning works, see [the docs](http://kubernetes.io/docs/user-guide/persistent-volumes/) or [this blog post](http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html).
## Quickstart
Choose some volume for your nfs-provisioner instance to store its state & data in and mount the volume at `/export` in `deploy/kubernetes/deployment.yaml`. It doesn't have to be a `hostPath` volume, it can e.g. be a PVC. Note that the volume must have a [supported file system](https://github.com/nfs-ganesha/nfs-ganesha/wiki/Fsalsupport#vfs) on it: any local filesystem on Linux is supported & NFS is not supported.
```yaml
...
volumeMounts:
- name: export-volume
mountPath: /export
volumes:
- name: export-volume
hostPath:
path: /tmp/nfs-provisioner
...
```
Choose a `provisioner` name for a `StorageClass` to specify and set it in `deploy/kubernetes/deployment.yaml`
```yaml
...
args:
- "-provisioner=example.com/nfs"
...
```
Create the deployment.
```console
$ kubectl create -f deploy/kubernetes/deployment.yaml
serviceaccount/nfs-provisioner created
service "nfs-provisioner" created
deployment "nfs-provisioner" created
```
Create `ClusterRole`, `ClusterRoleBinding`, `Role` and `RoleBinding` (this is necessary if you use RBAC authorization on your cluster, which is the default for newer kubernetes versions).
```console
$ kubectl create -f deploy/kubernetes/rbac.yaml
clusterrole.rbac.authorization.k8s.io/nfs-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-provisioner created
```
Create a `StorageClass` named "example-nfs" with `provisioner: example.com/nfs`.
```console
$ kubectl create -f deploy/kubernetes/class.yaml
storageclass "example-nfs" created
```
Create a `PersistentVolumeClaim` with annotation `volume.beta.kubernetes.io/storage-class: "example-nfs"`
```console
$ kubectl create -f deploy/kubernetes/claim.yaml
persistentvolumeclaim "nfs" created
```
A `PersistentVolume` is provisioned for the `PersistentVolumeClaim`. Now the claim can be consumed by some pod(s) and the backing NFS storage read from or written to.
```console
$ kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pvc-dce84888-7a9d-11e6-b1ee-5254001e0c1b 1Mi RWX Delete Bound default/nfs 23s
```
Deleting the `PersistentVolumeClaim` will cause the provisioner to delete the `PersistentVolume` and its data.
Deleting the provisioner deployment will cause any outstanding `PersistentVolumes` to become unusable for as long as the provisioner is gone.
## Running
To deploy nfs-provisioner on a Kubernetes cluster see [Deployment](docs/deployment.md).
To use nfs-provisioner once it is deployed see [Usage](docs/usage.md).
## [Changelog](CHANGELOG.md)
Releases done here in external-storage will not have corresponding git tags (external-storage's git tags are reserved for versioning the library), so to keep track of releases check this README, the [changelog](CHANGELOG.md), or [Quay](https://quay.io/repository/kubernetes_incubator/nfs-provisioner)
## Writing your own
Go [here](../docs/demo/hostpath-provisioner) for an example of how to write your own out-of-tree dynamic provisioner.
## Roadmap
This is still alpha/experimental and will change to reflect the [out-of-tree dynamic provisioner proposal](https://github.com/kubernetes/kubernetes/pull/30285)
## Community, discussion, contribution, and support
Learn how to engage with the Kubernetes community on the [community page](http://kubernetes.io/community/).
You can reach the maintainers of this project at:
- Slack: #sig-storage
## Kubernetes Incubator
This is a [Kubernetes Incubator project](https://github.com/kubernetes/community/blob/master/incubator.md). The project was established 2016-11-15. The incubator team for the project is:
- Sponsor: Clayton (@smarterclayton)
- Champion: Brad (@childsb)
- SIG: sig-storage
### Code of conduct
Participation in the Kubernetes community is governed by the [Kubernetes Code of Conduct](code-of-conduct.md).
nfs/RELEASE.md 0 → 100644
# Release Process
nfs-provisioner is released on an as-needed basis. The process is as follows:
1. An issue is proposing a new release with a changelog since the last release
1. An OWNER runs `make test` to make sure tests pass
1. An OWNER runs `git tag -a $VERSION` and inserts the changelog and pushes the tag with `git push $VERSION`
1. An OWNER runs `make push` to build and push the image
1. The release issue is closed
nfs/cmd/nfs-provisioner/main.go 0 → 100644
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
import (
"flag"
"strings"
"time"
"github.com/golang/glog"
"github.com/kubernetes-incubator/external-storage/nfs/pkg/server"
vol "github.com/kubernetes-incubator/external-storage/nfs/pkg/volume"
"github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/controller"
"k8s.io/apimachinery/pkg/util/validation"
"k8s.io/apimachinery/pkg/util/validation/field"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
)
var (
provisioner = flag.String("provisioner", "example.com/nfs", "Name of the provisioner. The provisioner will only provision volumes for claims that request a StorageClass with a provisioner field set equal to this name.")
master = flag.String("master", "", "Master URL to build a client config from. Either this or kubeconfig needs to be set if the provisioner is being run out of cluster.")
kubeconfig = flag.String("kubeconfig", "", "Absolute path to the kubeconfig file. Either this or master needs to be set if the provisioner is being run out of cluster.")
runServer = flag.Bool("run-server", true, "If the provisioner is responsible for running the NFS server, i.e. starting and stopping NFS Ganesha. Default true.")
useGanesha = flag.Bool("use-ganesha", true, "If the provisioner will create volumes using NFS Ganesha (D-Bus method calls) as opposed to using the kernel NFS server ('exportfs'). If run-server is true, this must be true. Default true.")
gracePeriod = flag.Uint("grace-period", 90, "NFS Ganesha grace period to use in seconds, from 0-180. If the server is not expected to survive restarts, i.e. it is running as a pod & its export directory is not persisted, this can be set to 0. Can only be set if both run-server and use-ganesha are true. Default 90.")
enableXfsQuota = flag.Bool("enable-xfs-quota", false, "If the provisioner will set xfs quotas for each volume it provisions. Requires that the directory it creates volumes in ('/export') is xfs mounted with option prjquota/pquota, and that it has the privilege to run xfs_quota. Default false.")
serverHostname = flag.String("server-hostname", "", "The hostname for the NFS server to export from. Only applicable when running out-of-cluster i.e. it can only be set if either master or kubeconfig are set. If unset, the first IP output by `hostname -i` is used.")
exportSubnet = flag.String("export-subnet", "*", "Subnet for NFS export to allow mount only from")
maxExports = flag.Int("max-exports", -1, "The maximum number of volumes to be exported by this provisioner. New claims will be ignored once this limit has been reached. A negative value is interpreted as 'unlimited'. Default -1.")
fsidDevice = flag.Bool("device-based-fsids", true, "If file system handles created by NFS Ganesha should be based on major/minor device IDs of the backing storage volume ('/export'). Default true.")
)
const (
exportDir = "/export"
ganeshaLog = "/export/ganesha.log"
ganeshaPid = "/var/run/ganesha.pid"
ganeshaConfig = "/export/vfs.conf"
)
func main() {
flag.Set("logtostderr", "true")
flag.Parse()
if errs := validateProvisioner(*provisioner, field.NewPath("provisioner")); len(errs) != 0 {
glog.Fatalf("Invalid provisioner specified: %v", errs)
}
glog.Infof("Provisioner %s specified", *provisioner)
if *runServer && !*useGanesha {
glog.Fatalf("Invalid flags specified: if run-server is true, use-ganesha must also be true.")
}
if *useGanesha && *exportSubnet != "*" {
glog.Warningf("If use-ganesha is true, there is no effect on export-subnet.")
}
if *gracePeriod != 90 && (!*runServer || !*useGanesha) {
glog.Fatalf("Invalid flags specified: custom grace period can only be set if both run-server and use-ganesha are true.")
} else if *gracePeriod > 180 && *runServer && *useGanesha {
glog.Fatalf("Invalid flags specified: custom grace period must be in the range 0-180")
}
// Create the client according to whether we are running in or out-of-cluster
outOfCluster := *master != "" || *kubeconfig != ""
if !outOfCluster && *serverHostname != "" {
glog.Fatalf("Invalid flags specified: if server-hostname is set, either master or kube-config must also be set.")
}
if *runServer {
glog.Infof("Setting up NFS server!")
err := server.Setup(ganeshaConfig, *gracePeriod, *fsidDevice)
if err != nil {
glog.Fatalf("Error setting up NFS server: %v", err)
}
go func() {
for {
// This blocks until server exits (presumably due to an error)
err = server.Run(ganeshaLog, ganeshaPid, ganeshaConfig)
if err != nil {
glog.Errorf("NFS server Exited Unexpectedly with err: %v", err)
}
// take a moment before trying to restart
time.Sleep(time.Second)
}
}()
// Wait for NFS server to come up before continuing provisioner process
time.Sleep(5 * time.Second)
}
var config *rest.Config
var err error
if outOfCluster {
config, err = clientcmd.BuildConfigFromFlags(*master, *kubeconfig)
} else {
config, err = rest.InClusterConfig()
}
if err != nil {
glog.Fatalf("Failed to create config: %v", err)
}
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
glog.Fatalf("Failed to create client: %v", err)
}
// The controller needs to know what the server version is because out-of-tree
// provisioners aren't officially supported until 1.5
serverVersion, err := clientset.Discovery().ServerVersion()
if err != nil {
glog.Fatalf("Error getting server version: %v", err)
}
// Create the provisioner: it implements the Provisioner interface expected by
// the controller
nfsProvisioner := vol.NewNFSProvisioner(exportDir, clientset, outOfCluster, *useGanesha, ganeshaConfig, *enableXfsQuota, *serverHostname, *maxExports, *exportSubnet)
// Start the provision controller which will dynamically provision NFS PVs
pc := controller.NewProvisionController(
clientset,
*provisioner,
nfsProvisioner,
serverVersion.GitVersion,
)
pc.Run(wait.NeverStop)
}
// validateProvisioner tests if provisioner is a valid qualified name.
// https://github.com/kubernetes/kubernetes/blob/release-1.4/pkg/apis/storage/validation/validation.go
func validateProvisioner(provisioner string, fldPath *field.Path) field.ErrorList {
allErrs := field.ErrorList{}
if len(provisioner) == 0 {
allErrs = append(allErrs, field.Required(fldPath, provisioner))
}
if len(provisioner) > 0 {
for _, msg := range validation.IsQualifiedName(strings.ToLower(provisioner)) {
allErrs = append(allErrs, field.Invalid(fldPath, provisioner, msg))
}
}
return allErrs
}
nfs/deploy/README.md 0 → 100644
Please see [Deployment](../docs/deployment.md) for how to deploy nfs-provisioner on a Kubernetes cluster using these files.
nfs/deploy/docker/Dockerfile 0 → 100644
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Modified from https://github.com/rootfs/nfs-ganesha-docker by Huamin Chen
FROM fedora:30 AS build
# Build ganesha from source, install it to /usr/local and a use multi stage build to have a smaller image
RUN dnf install -y tar gcc cmake autoconf libtool bison flex make gcc-c++ krb5-devel dbus-devel jemalloc-devel libnfsidmap-devel libnsl2-devel userspace-rcu-devel patch libblkid-devel
RUN curl -L https://github.com/nfs-ganesha/nfs-ganesha/archive/V2.8.2.tar.gz | tar zx \
&& curl -L https://github.com/nfs-ganesha/ntirpc/archive/v1.8.0.tar.gz | tar zx \
&& rm -r nfs-ganesha-2.8.2/src/libntirpc \
&& mv ntirpc-1.8.0 nfs-ganesha-2.8.2/src/libntirpc
WORKDIR /nfs-ganesha-2.8.2
RUN mkdir -p /usr/local \
&& cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_CONFIG=vfs_only -DCMAKE_INSTALL_PREFIX=/usr/local src/ \
&& make \
&& make install
RUN mkdir -p /ganesha-extra \
&& mkdir -p /ganesha-extra/etc/dbus-1/system.d \
&& cp src/scripts/ganeshactl/org.ganesha.nfsd.conf /ganesha-extra/etc/dbus-1/system.d/
FROM registry.fedoraproject.org/fedora-minimal:30 AS run
RUN microdnf install -y libblkid userspace-rcu dbus-x11 rpcbind hostname nfs-utils xfsprogs jemalloc libnfsidmap && microdnf clean all
RUN mkdir -p /var/run/dbus \
&& mkdir -p /export
# add libs from /usr/local/lib64
RUN echo /usr/local/lib64 > /etc/ld.so.conf.d/local_libs.conf
# do not ask systemd for user IDs or groups (slows down dbus-daemon start)
RUN sed -i s/systemd// /etc/nsswitch.conf
COPY --from=build /usr/local /usr/local/
COPY --from=build /ganesha-extra /
COPY nfs-provisioner /nfs-provisioner
# run ldconfig after libs have been copied
RUN ldconfig
# expose mountd 20048/tcp and nfsd 2049/tcp and rpcbind 111/tcp 111/udp
EXPOSE 2049/tcp 20048/tcp 111/tcp 111/udp
ENTRYPOINT ["/nfs-provisioner"]
nfs/deploy/docker/config 0 → 100644
apiVersion: v1
clusters:
- cluster:
insecure-skip-tls-verify: true
server: http://0.0.0.0:8080
name: local
contexts:
- context:
cluster: local
user: ""
name: local
current-context: local
kind: Config
preferences: {}
users: []
nfs/deploy/kubernetes/claim.yaml 0 → 100644
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nfs
annotations:
volume.beta.kubernetes.io/storage-class: "example-nfs"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Mi
nfs/deploy/kubernetes/class.yaml 0 → 100644
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: example-nfs
provisioner: example.com/nfs
mountOptions:
- vers=4.1
nfs/deploy/kubernetes/deployment.yaml 0 → 100644
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
---
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
ports:
- name: nfs
port: 2049
- name: mountd
port: 20048
- name: rpcbind
port: 111
- name: rpcbind-udp
port: 111
protocol: UDP
selector:
app: nfs-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
containers:
- name: nfs-provisioner
image: quay.io/kubernetes_incubator/nfs-provisioner:latest
ports:
- name: nfs
containerPort: 2049
- name: mountd
containerPort: 20048
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
securityContext:
capabilities:
add:
- DAC_READ_SEARCH
- SYS_RESOURCE
args:
- "-provisioner=example.com/nfs"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: SERVICE_NAME
value: nfs-provisioner
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /export
volumes:
- name: export-volume
hostPath:
path: /srv
nfs/deploy/kubernetes/pod.yaml 0 → 100644
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
---
kind: Pod
apiVersion: v1
metadata:
name: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
containers:
- name: nfs-provisioner
image: quay.io/kubernetes_incubator/nfs-provisioner:latest
ports:
- name: nfs
containerPort: 2049
- name: mountd
containerPort: 20048
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
securityContext:
capabilities:
add:
- DAC_READ_SEARCH
args:
- "-provisioner=example.com/nfs"
- "-grace-period=0"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /export
volumes:
- name: export-volume
emptyDir: {}
nfs/deploy/kubernetes/psp.yaml 0 → 100644
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: nfs-provisioner
spec:
fsGroup:
rule: RunAsAny
allowedCapabilities:
- DAC_READ_SEARCH
- SYS_RESOURCE
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- secret
- hostPath
nfs/deploy/kubernetes/rbac.yaml 0 → 100644
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services", "endpoints"]
verbs: ["get"]
- apiGroups: ["extensions"]
resources: ["podsecuritypolicies"]
resourceNames: ["nfs-provisioner"]
verbs: ["use"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: ClusterRole
name: nfs-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-provisioner
apiGroup: rbac.authorization.k8s.io
nfs/deploy/kubernetes/read-pod.yaml 0 → 100644
kind: Pod
apiVersion: v1
metadata:
name: read-pod
spec:
containers:
- name: read-pod
image: gcr.io/google_containers/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "test -f /mnt/SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: nfs-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: nfs
nfs/deploy/kubernetes/scc.yaml 0 → 100644
allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegedContainer: false
allowedCapabilities:
- DAC_READ_SEARCH
- SYS_RESOURCE
apiVersion: v1
defaultAddCapabilities: null
fsGroup:
type: MustRunAs
kind: SecurityContextConstraints
metadata:
annotations: null
name: nfs-provisioner
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities:
- KILL
- MKNOD
- SYS_CHROOT
runAsUser:
type: RunAsAny
seLinuxContext:
type: MustRunAs
supplementalGroups:
type: RunAsAny
volumes:
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- secret
nfs/deploy/kubernetes/statefulset.yaml 0 → 100644
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
---
kind: Service
apiVersion: v1
metadata:
name: nfs-provisioner
labels:
app: nfs-provisioner
spec:
ports:
- name: nfs
port: 2049
- name: mountd
port: 20048
- name: rpcbind
port: 111
- name: rpcbind-udp
port: 111
protocol: UDP
selector:
app: nfs-provisioner
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
serviceName: "nfs-provisioner"
replicas: 1
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
terminationGracePeriodSeconds: 10
containers:
- name: nfs-provisioner
image: quay.io/kubernetes_incubator/nfs-provisioner:latest
ports:
- name: nfs
containerPort: 2049
- name: mountd
containerPort: 20048
- name: rpcbind
containerPort: 111
- name: rpcbind-udp
containerPort: 111
protocol: UDP
securityContext:
capabilities:
add:
- DAC_READ_SEARCH
- SYS_RESOURCE
args:
- "-provisioner=example.com/nfs"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: SERVICE_NAME
value: nfs-provisioner
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: export-volume
mountPath: /export
volumes:
- name: export-volume
hostPath:
path: /srv
nfs/deploy/kubernetes/write-pod.yaml 0 → 100644
kind: Pod
apiVersion: v1
metadata:
name: write-pod
spec:
containers:
- name: write-pod
image: gcr.io/google_containers/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: nfs-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: nfs
nfs/docs/usage.md 0 → 100644
## Usage
The nfs-provisioner has been deployed and is now watching for claims it should provision volumes for. No such claims can exist until a properly configured `StorageClass` for claims to request is created.
Edit the `provisioner` field in `deploy/kubernetes/class.yaml` to be the provisioner's name. Configure the `parameters`.
### Parameters
* `gid`: `"none"` or a [supplemental group](http://kubernetes.io/docs/user-guide/security-context/) like `"1001"`. NFS shares will be created with permissions such that pods running with the supplemental group can read & write to the share, but non-root pods without the supplemental group cannot. Pods running as root can read & write to shares regardless of the setting here, unless the `rootSquash` parameter is set true. If set to `"none"`, anybody root or non-root can write to the share. Default (if omitted) `"none"`.
* `rootSquash`: `"true"` or `"false"`. Whether to squash root users by adding the NFS Ganesha root_id_squash or kernel root_squash option to each export. Default `"false"`.
* **Deprecated. Use StorageClass.mountOptions instead.** `mountOptions`: a comma separated list of [mount options](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options) for every PV of this class to be mounted with. The list is inserted directly into every PV's mount options annotation/field without any validation. Default blank `""`.
Name the `StorageClass` however you like; the name is how claims will request this class. Create the class.
```
$ kubectl create -f deploy/kubernetes/class.yaml
storageclass "example-nfs" created
```
Now if everything is working correctly, when you create a claim requesting the class you just created, the provisioner will automatically create a volume.
Edit the `volume.beta.kubernetes.io/storage-class` annotation in `deploy/kubernetes/claim.yaml` to be the name of the class. Create the claim.
```
$ kubectl create -f deploy/kubernetes/claim.yaml
persistentvolumeclaim "nfs" created
```
The nfs-provisioner provisions a PV for the PVC you just created. Its reclaim policy is Delete, so it and its backing storage will be deleted by the provisioner when the PVC is deleted.
```
$ kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pvc-dce84888-7a9d-11e6-b1ee-5254001e0c1b 1Mi RWX Delete Bound default/nfs 23s
```
A pod can consume the PVC and write to the backing NFS share. Create a pod to test this.
```
$ kubectl create -f deploy/kubernetes/write-pod.yaml
pod "write-pod" created
$ kubectl get pod --show-all
nfs-provisioner 1/1 Running 0 31s
write-pod 0/1 Completed 0 41s
```
Once you are done with the PVC, delete it and the provisioner will delete the PV and its backing storage.
```
$ kubectl delete pod write-pod
pod "write-pod" deleted
$ kubectl delete pvc nfs
persistentvolumeclaim "nfs" deleted
$ kubectl get pv
```
Note that deleting or stopping a provisioner won't delete the `PersistentVolume` objects it created.
If at any point things don't work correctly, check the provisioner's logs using `kubectl logs` and look for events in the PVs and PVCs using `kubectl describe`.
### Using as default
The provisioner can be used as the default storage provider, meaning claims that don't request a `StorageClass` get volumes provisioned for them by the provisioner by default. To set as the default a `StorageClass` that specifies the provisioner, turn on the `DefaultStorageClass` admission-plugin and add the `storageclass.beta.kubernetes.io/is-default-class` annotation to the class. See http://kubernetes.io/docs/user-guide/persistent-volumes/#class-1 for more information.
nfs/go.mod 0 → 100644
module github.com/kubernetes-incubator/external-storage/nfs
go 1.12
require (
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/guelfey/go.dbus v0.0.0-20131113121618-f6a3a2366cc3
github.com/imdario/mergo v0.3.7 // indirect
github.com/kubernetes-sigs/sig-storage-lib-external-provisioner v4.0.0+incompatible
github.com/miekg/dns v1.1.15 // indirect
github.com/prometheus/client_golang v1.1.0 // indirect
golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4 // indirect
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 // indirect
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 // indirect
k8s.io/api v0.0.0-20190814101207-0772a1bdf941
k8s.io/apimachinery v0.0.0-20190814100815-533d101be9a6
k8s.io/client-go v0.0.0-20190816061517-44c2c549a534
k8s.io/utils v0.0.0-20190809000727-6c36bc71fc4a // indirect
sigs.k8s.io/sig-storage-lib-external-provisioner v4.0.0+incompatible // indirect
)
nfs/go.sum 0 → 100644
This diff is collapsed. Click to expand it.
nfs/pkg/server/server.go 0 → 100644
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package server
import (
"fmt"
"io/ioutil"
"os"
"os/exec"
"regexp"
"strings"
"syscall"
"github.com/golang/glog"
)
var defaultGaneshaConfigContents = []byte(`
###################################################
#
# EXPORT
#
# To function, all that is required is an EXPORT
#
# Define the absolute minimal export
#
###################################################
EXPORT
{
# Export Id (mandatory, each EXPORT must have a unique Export_Id)
Export_Id = 0;
# Exported path (mandatory)
Path = /nonexistent;
# Pseudo Path (required for NFS v4)
Pseudo = /nonexistent;
# Required for access (default is None)
# Could use CLIENT blocks instead
Access_Type = RW;
# Exporting FSAL
FSAL {
Name = VFS;
}
}
NFS_Core_Param
{
MNT_Port = 20048;
fsid_device = true;
}
NFSV4
{
Grace_Period = 90;
}
`)
// Setup sets up various prerequisites and settings for the server. If an error
// is encountered at any point it returns it instantly
func Setup(ganeshaConfig string, gracePeriod uint, fsidDevice bool) error {
// Start rpcbind if it is not started yet
cmd := exec.Command("/usr/sbin/rpcinfo", "127.0.0.1")
if err := cmd.Run(); err != nil {
cmd = exec.Command("/usr/sbin/rpcbind", "-w")
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("Starting rpcbind failed with error: %v, output: %s", err, out)
}
}
cmd = exec.Command("/usr/sbin/rpc.statd")
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("rpc.statd failed with error: %v, output: %s", err, out)
}
// Start dbus, needed for ganesha dynamic exports
cmd = exec.Command("dbus-daemon", "--system")
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("dbus-daemon failed with error: %v, output: %s", err, out)
}
err := setRlimitNOFILE()
if err != nil {
glog.Warningf("Error setting RLIMIT_NOFILE, there may be 'Too many open files' errors later: %v", err)
}
// Use defaultGaneshaConfigContents if the ganeshaConfig doesn't exist yet
if _, err = os.Stat(ganeshaConfig); os.IsNotExist(err) {
err = ioutil.WriteFile(ganeshaConfig, defaultGaneshaConfigContents, 0600)
if err != nil {
return fmt.Errorf("error writing ganesha config %s: %v", ganeshaConfig, err)
}
}
err = setGracePeriod(ganeshaConfig, gracePeriod)
if err != nil {
return fmt.Errorf("error setting grace period to ganesha config: %v", err)
}
err = setFsidDevice(ganeshaConfig, fsidDevice)
if err != nil {
return fmt.Errorf("error setting fsid device to ganesha config: %v", err)
}
return nil
}
// Run : run the NFS server in the foreground until it exits
// Ideally, it should never exit when run in foreground mode
// We force foreground to allow the provisioner process to restart
// the server if it crashes - daemonization prevents us from using Wait()
// for this purpose
func Run(ganeshaLog, ganeshaPid, ganeshaConfig string) error {
// Start ganesha.nfsd
glog.Infof("Running NFS server!")
cmd := exec.Command("ganesha.nfsd", "-F", "-L", ganeshaLog, "-p", ganeshaPid, "-f", ganeshaConfig)
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("ganesha.nfsd failed with error: %v, output: %s", err, out)
}
return nil
}
func setRlimitNOFILE() error {
var rlimit syscall.Rlimit
err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rlimit)
if err != nil {
return fmt.Errorf("error getting RLIMIT_NOFILE: %v", err)
}
glog.Infof("starting RLIMIT_NOFILE rlimit.Cur %d, rlimit.Max %d", rlimit.Cur, rlimit.Max)
rlimit.Max = 1024 * 1024
rlimit.Cur = 1024 * 1024
err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rlimit)
if err != nil {
return err
}
err = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rlimit)
if err != nil {
return fmt.Errorf("error getting RLIMIT_NOFILE: %v", err)
}
glog.Infof("ending RLIMIT_NOFILE rlimit.Cur %d, rlimit.Max %d", rlimit.Cur, rlimit.Max)
return nil
}
func setFsidDevice(ganeshaConfig string, fsidDevice bool) error {
newLine := fmt.Sprintf("fsid_device = %t;", fsidDevice)
re := regexp.MustCompile("fsid_device = (true|false);")
read, err := ioutil.ReadFile(ganeshaConfig)
if err != nil {
return err
}
oldLine := re.Find(read)
if oldLine == nil {
// fsid_device line not there, append it after MNT_Port
re := regexp.MustCompile("MNT_Port = 20048;")
mntPort := re.Find(read)
block := "MNT_Port = 20048;\n" +
"\t" + newLine
replaced := strings.Replace(string(read), string(mntPort), block, -1)
err = ioutil.WriteFile(ganeshaConfig, []byte(replaced), 0)
if err != nil {
return err
}
} else {
// fsid_device there, just replace it
replaced := strings.Replace(string(read), string(oldLine), newLine, -1)
err = ioutil.WriteFile(ganeshaConfig, []byte(replaced), 0)
if err != nil {
return err
}
}
return nil
}
func setGracePeriod(ganeshaConfig string, gracePeriod uint) error {
if gracePeriod > 180 {
return fmt.Errorf("grace period cannot be greater than 180")
}
newLine := fmt.Sprintf("Grace_Period = %d;", gracePeriod)
re := regexp.MustCompile("Grace_Period = [0-9]+;")
read, err := ioutil.ReadFile(ganeshaConfig)
if err != nil {
return err
}
oldLine := re.Find(read)
var file *os.File
if oldLine == nil {
// Grace_Period line not there, append the whole NFSV4 block.
file, err = os.OpenFile(ganeshaConfig, os.O_APPEND|os.O_WRONLY, 0600)
if err != nil {
return err
}
defer file.Close()
block := "\nNFSV4\n{\n" +
"\t" + newLine + "\n" +
"}\n"
if _, err = file.WriteString(block); err != nil {
return err
}
file.Sync()
} else {
// Grace_Period line there, just replace it
replaced := strings.Replace(string(read), string(oldLine), newLine, -1)
err = ioutil.WriteFile(ganeshaConfig, []byte(replaced), 0)
if err != nil {
return err
}
}
return nil
}
// Stop stops the NFS server.
func Stop() {
// /bin/dbus-send --system --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.shutdown
}
nfs/pkg/volume/delete.go 0 → 100644
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package volume
import (
"fmt"
"os"
"path"
"strconv"
"github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/controller"
"k8s.io/api/core/v1"
)
// Delete removes the directory that was created by Provision backing the given
// PV and removes its export from the NFS server.
func (p *nfsProvisioner) Delete(volume *v1.PersistentVolume) error {
// Ignore the call if this provisioner was not the one to provision the
// volume. It doesn't even attempt to delete it, so it's neither a success
// (nil error) nor failure (any other error)
provisioned, err := p.provisioned(volume)
if err != nil {
return fmt.Errorf("error determining if this provisioner was the one to provision volume %q: %v", volume.Name, err)
}
if !provisioned {
strerr := fmt.Sprintf("this provisioner id %s didn't provision volume %q and so can't delete it; id %s did & can", p.identity, volume.Name, volume.Annotations[annProvisionerID])
return &controller.IgnoredError{Reason: strerr}
}
err = p.deleteDirectory(volume)
if err != nil {
return fmt.Errorf("error deleting volume's backing path: %v", err)
}
err = p.deleteExport(volume)
if err != nil {
return fmt.Errorf("deleted the volume's backing path but error deleting export: %v", err)
}
err = p.deleteQuota(volume)
if err != nil {
return fmt.Errorf("deleted the volume's backing path & export but error deleting quota: %v", err)
}
return nil
}
func (p *nfsProvisioner) provisioned(volume *v1.PersistentVolume) (bool, error) {
provisionerID, ok := volume.Annotations[annProvisionerID]
if !ok {
return false, fmt.Errorf("PV doesn't have an annotation %s", annProvisionerID)
}
return provisionerID == string(p.identity), nil
}
func (p *nfsProvisioner) deleteDirectory(volume *v1.PersistentVolume) error {
path := path.Join(p.exportDir, volume.ObjectMeta.Name)
if _, err := os.Stat(path); os.IsNotExist(err) {
return nil
}
if err := os.RemoveAll(path); err != nil {
return err
}
return nil
}
func (p *nfsProvisioner) deleteExport(volume *v1.PersistentVolume) error {
block, exportID, err := getBlockAndID(volume, annExportBlock, annExportID)
if err != nil {
return fmt.Errorf("error getting block &/or id from annotations: %v", err)
}
if err := p.exporter.RemoveExportBlock(block, uint16(exportID)); err != nil {
return fmt.Errorf("error removing the export from the config file: %v", err)
}
if err := p.exporter.Unexport(volume); err != nil {
return fmt.Errorf("removed export from the config file but error unexporting it: %v", err)
}
return nil
}
func (p *nfsProvisioner) deleteQuota(volume *v1.PersistentVolume) error {
block, projectID, err := getBlockAndID(volume, annProjectBlock, annProjectID)
if err != nil {
return fmt.Errorf("error getting block &/or id from annotations: %v", err)
}
if err := p.quotaer.RemoveProject(block, uint16(projectID)); err != nil {
return fmt.Errorf("error removing the quota project from the projects file: %v", err)
}
if err := p.quotaer.UnsetQuota(); err != nil {
return fmt.Errorf("removed quota project from the project file but error unsetting the quota: %v", err)
}
return nil
}
func getBlockAndID(volume *v1.PersistentVolume, annBlock, annID string) (string, uint16, error) {
block, ok := volume.Annotations[annBlock]
if !ok {
return "", 0, fmt.Errorf("PV doesn't have an annotation with key %s", annBlock)
}
idStr, ok := volume.Annotations[annID]
if !ok {
return "", 0, fmt.Errorf("PV doesn't have an annotation %s", annID)
}
id, _ := strconv.ParseUint(idStr, 10, 16)
return block, uint16(id), nil
}
nfs/pkg/volume/export.go 0 → 100644
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package volume
import (
"fmt"
"os"
"os/exec"
"regexp"
"strconv"
"sync"
"github.com/golang/glog"
"github.com/guelfey/go.dbus"
"k8s.io/api/core/v1"
)
type exporter interface {
CanExport(int) bool
AddExportBlock(string, bool, string) (string, uint16, error)
RemoveExportBlock(string, uint16) error
Export(string) error
Unexport(*v1.PersistentVolume) error
}
type exportBlockCreator interface {
CreateExportBlock(string, string, bool, string) string
}
type exportMap struct {
// Map to track used exportIDs. Each ganesha export needs a unique fsid and
// Export_Id, each kernel a unique fsid. Assign each export an exportID and
// use it as both fsid and Export_Id.
exportIDs map[uint16]bool
}
func (e *exportMap) CanExport(limit int) bool {
if limit < 0 {
return true
}
totalExports := len(e.exportIDs)
return totalExports < limit
}
type genericExporter struct {
*exportMap
ebc exportBlockCreator
config string
mapMutex *sync.Mutex
fileMutex *sync.Mutex
}
func newGenericExporter(ebc exportBlockCreator, config string, re *regexp.Regexp) *genericExporter {
if _, err := os.Stat(config); os.IsNotExist(err) {
glog.Fatalf("config %s does not exist!", config)
}
exportIDs, err := getExistingIDs(config, re)
if err != nil {
glog.Errorf("error while populating exportIDs map, there may be errors exporting later if exportIDs are reused: %v", err)
}
return &genericExporter{
exportMap: &exportMap{
exportIDs: exportIDs,
},
ebc: ebc,
config: config,
mapMutex: &sync.Mutex{},
fileMutex: &sync.Mutex{},
}
}
func (e *genericExporter) AddExportBlock(path string, rootSquash bool, exportSubnet string) (string, uint16, error) {
exportID := generateID(e.mapMutex, e.exportIDs)
exportIDStr := strconv.FormatUint(uint64(exportID), 10)
block := e.ebc.CreateExportBlock(exportIDStr, path, rootSquash, exportSubnet)
// Add the export block to the config file
if err := addToFile(e.fileMutex, e.config, block); err != nil {
deleteID(e.mapMutex, e.exportIDs, exportID)
return "", 0, fmt.Errorf("error adding export block %s to config %s: %v", block, e.config, err)
}
return block, exportID, nil
}
func (e *genericExporter) RemoveExportBlock(block string, exportID uint16) error {
deleteID(e.mapMutex, e.exportIDs, exportID)
return removeFromFile(e.fileMutex, e.config, block)
}
type ganeshaExporter struct {
genericExporter
}
var _ exporter = &ganeshaExporter{}
func newGaneshaExporter(ganeshaConfig string) exporter {
return &ganeshaExporter{
genericExporter: *newGenericExporter(&ganeshaExportBlockCreator{}, ganeshaConfig, regexp.MustCompile("Export_Id = ([0-9]+);")),
}
}
// Export exports the given directory using NFS Ganesha, assuming it is running
// and can be connected to using D-Bus.
func (e *ganeshaExporter) Export(path string) error {
// Call AddExport using dbus
conn, err := dbus.SystemBus()
if err != nil {
return fmt.Errorf("error getting dbus session bus: %v", err)
}
obj := conn.Object("org.ganesha.nfsd", "/org/ganesha/nfsd/ExportMgr")
call := obj.Call("org.ganesha.nfsd.exportmgr.AddExport", 0, e.config, fmt.Sprintf("export(path = %s)", path))
if call.Err != nil {
return fmt.Errorf("error calling org.ganesha.nfsd.exportmgr.AddExport: %v", call.Err)
}
return nil
}
func (e *ganeshaExporter) Unexport(volume *v1.PersistentVolume) error {
ann, ok := volume.Annotations[annExportID]
if !ok {
return fmt.Errorf("PV doesn't have an annotation %s, can't remove the export from the server", annExportID)
}
exportID, _ := strconv.ParseUint(ann, 10, 16)
// Call RemoveExport using dbus
conn, err := dbus.SystemBus()
if err != nil {
return fmt.Errorf("error getting dbus session bus: %v", err)
}
obj := conn.Object("org.ganesha.nfsd", "/org/ganesha/nfsd/ExportMgr")
call := obj.Call("org.ganesha.nfsd.exportmgr.RemoveExport", 0, uint16(exportID))
if call.Err != nil {
return fmt.Errorf("error calling org.ganesha.nfsd.exportmgr.RemoveExport: %v", call.Err)
}
return nil
}
type ganeshaExportBlockCreator struct{}
var _ exportBlockCreator = &ganeshaExportBlockCreator{}
// CreateBlock creates the text block to add to the ganesha config file.
func (e *ganeshaExportBlockCreator) CreateExportBlock(exportID, path string, rootSquash bool, exportSubnet string) string {
squash := "no_root_squash"
if rootSquash {
squash = "root_id_squash"
}
return "\nEXPORT\n{\n" +
"\tExport_Id = " + exportID + ";\n" +
"\tPath = " + path + ";\n" +
"\tPseudo = " + path + ";\n" +
"\tAccess_Type = RW;\n" +
"\tSquash = " + squash + ";\n" +
"\tSecType = sys;\n" +
"\tFilesystem_id = " + exportID + "." + exportID + ";\n" +
"\tFSAL {\n\t\tName = VFS;\n\t}\n}\n"
}
type kernelExporter struct {
genericExporter
}
var _ exporter = &kernelExporter{}
func newKernelExporter() exporter {
return &kernelExporter{
genericExporter: *newGenericExporter(&kernelExportBlockCreator{}, "/etc/exports", regexp.MustCompile("fsid=([0-9]+)")),
}
}
// Export exports all directories listed in /etc/exports
func (e *kernelExporter) Export(_ string) error {
// Execute exportfs
cmd := exec.Command("exportfs", "-r")
out, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("exportfs -r failed with error: %v, output: %s", err, out)
}
return nil
}
func (e *kernelExporter) Unexport(volume *v1.PersistentVolume) error {
// Execute exportfs
cmd := exec.Command("exportfs", "-r")
out, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("exportfs -r failed with error: %v, output: %s", err, out)
}
return nil
}
type kernelExportBlockCreator struct{}
var _ exportBlockCreator = &kernelExportBlockCreator{}
// CreateBlock creates the text block to add to the /etc/exports file.
func (e *kernelExportBlockCreator) CreateExportBlock(exportID, path string, rootSquash bool, exportSubnet string) string {
squash := "no_root_squash"
if rootSquash {
squash = "root_squash"
}
return "\n" + path + " " + exportSubnet + "(rw,insecure," + squash + ",fsid=" + exportID + ")\n"
}
nfs/pkg/volume/quota.go 0 → 100644
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package volume
import (
"fmt"
"io/ioutil"
"os"
"os/exec"
"path"
"regexp"
"strconv"
"strings"
"sync"
"github.com/golang/glog"
"github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/mount"
)
type quotaer interface {
AddProject(string, string) (string, uint16, error)
RemoveProject(string, uint16) error
SetQuota(uint16, string, string) error
UnsetQuota() error
}
type xfsQuotaer struct {
xfsPath string
// The file where we store mappings between project ids and directories, and
// each project's quota limit information, for backup.
// Similar to http://man7.org/linux/man-pages/man5/projects.5.html
projectsFile string
projectIDs map[uint16]bool
mapMutex *sync.Mutex
fileMutex *sync.Mutex
}
var _ quotaer = &xfsQuotaer{}
func newXfsQuotaer(xfsPath string) (*xfsQuotaer, error) {
if _, err := os.Stat(xfsPath); os.IsNotExist(err) {
return nil, fmt.Errorf("xfs path %s does not exist", xfsPath)
}
isXfs, err := isXfs(xfsPath)
if err != nil {
return nil, fmt.Errorf("error checking if xfs path %s is an XFS filesystem: %v", xfsPath, err)
}
if !isXfs {
return nil, fmt.Errorf("xfs path %s is not an XFS filesystem", xfsPath)
}
entry, err := getMountEntry(path.Clean(xfsPath), "xfs")
if err != nil {
return nil, err
}
if !strings.Contains(entry.VfsOpts, "pquota") && !strings.Contains(entry.VfsOpts, "prjquota") {
return nil, fmt.Errorf("xfs path %s was not mounted with pquota nor prjquota", xfsPath)
}
_, err = exec.LookPath("xfs_quota")
if err != nil {
return nil, err
}
projectsFile := path.Join(xfsPath, "projects")
projectIDs := map[uint16]bool{}
_, err = os.Stat(projectsFile)
if os.IsNotExist(err) {
file, cerr := os.Create(projectsFile)
if cerr != nil {
return nil, fmt.Errorf("error creating xfs projects file %s: %v", projectsFile, cerr)
}
file.Close()
} else {
re := regexp.MustCompile("(?m:^([0-9]+):/.+$)")
projectIDs, err = getExistingIDs(projectsFile, re)
if err != nil {
glog.Errorf("error while populating projectIDs map, there may be errors setting quotas later if projectIDs are reused: %v", err)
}
}
xfsQuotaer := &xfsQuotaer{
xfsPath: xfsPath,
projectsFile: projectsFile,
projectIDs: projectIDs,
mapMutex: &sync.Mutex{},
fileMutex: &sync.Mutex{},
}
err = xfsQuotaer.restoreQuotas()
if err != nil {
return nil, fmt.Errorf("error restoring quotas from projects file %s: %v", projectsFile, err)
}
return xfsQuotaer, nil
}
func isXfs(xfsPath string) (bool, error) {
cmd := exec.Command("stat", "-f", "-c", "%T", xfsPath)
out, err := cmd.Output()
if err != nil {
return false, err
}
if strings.TrimSpace(string(out)) != "xfs" {
return false, nil
}
return true, nil
}
func getMountEntry(mountpoint, fstype string) (*mount.Info, error) {
entries, err := mount.GetMounts()
if err != nil {
return nil, err
}
for _, e := range entries {
if e.Mountpoint == mountpoint && e.Fstype == fstype {
return e, nil
}
}
return nil, fmt.Errorf("mount entry for mountpoint %s, fstype %s not found", mountpoint, fstype)
}
func (q *xfsQuotaer) restoreQuotas() error {
read, err := ioutil.ReadFile(q.projectsFile)
if err != nil {
return err
}
re := regexp.MustCompile("(?m:\n^([0-9]+):(.+):(.+)$\n)")
matches := re.FindAllSubmatch(read, -1)
for _, match := range matches {
projectID, _ := strconv.ParseUint(string(match[1]), 10, 16)
directory := string(match[2])
bhard := string(match[3])
// If directory referenced by projects file no longer exists, don't set a
// quota for it: will fail
if _, err := os.Stat(directory); os.IsNotExist(err) {
q.RemoveProject(string(match[0]), uint16(projectID))
continue
}
if err := q.SetQuota(uint16(projectID), directory, bhard); err != nil {
return fmt.Errorf("error restoring quota for directory %s: %v", directory, err)
}
}
return nil
}
func (q *xfsQuotaer) AddProject(directory, bhard string) (string, uint16, error) {
projectID := generateID(q.mapMutex, q.projectIDs)
projectIDStr := strconv.FormatUint(uint64(projectID), 10)
// Store project:directory mapping and also project's quota info
block := "\n" + projectIDStr + ":" + directory + ":" + bhard + "\n"
// Add the project block to the projects file
if err := addToFile(q.fileMutex, q.projectsFile, block); err != nil {
deleteID(q.mapMutex, q.projectIDs, projectID)
return "", 0, fmt.Errorf("error adding project block %s to projects file %s: %v", block, q.projectsFile, err)
}
// Specify the new project
cmd := exec.Command("xfs_quota", "-x", "-c", fmt.Sprintf("project -s -p %s %s", directory, projectIDStr), q.xfsPath)
out, err := cmd.CombinedOutput()
if err != nil {
deleteID(q.mapMutex, q.projectIDs, projectID)
removeFromFile(q.fileMutex, q.projectsFile, block)
return "", 0, fmt.Errorf("xfs_quota failed with error: %v, output: %s", err, out)
}
return block, projectID, nil
}
func (q *xfsQuotaer) RemoveProject(block string, projectID uint16) error {
deleteID(q.mapMutex, q.projectIDs, projectID)
return removeFromFile(q.fileMutex, q.projectsFile, block)
}
func (q *xfsQuotaer) SetQuota(projectID uint16, directory, bhard string) error {
if !q.projectIDs[projectID] {
return fmt.Errorf("project with id %v has not been added", projectID)
}
projectIDStr := strconv.FormatUint(uint64(projectID), 10)
cmd := exec.Command("xfs_quota", "-x", "-c", fmt.Sprintf("limit -p bhard=%s %s", bhard, projectIDStr), q.xfsPath)
out, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("xfs_quota failed with error: %v, output: %s", err, out)
}
return nil
}
func (q *xfsQuotaer) UnsetQuota() error {
return nil
}
type dummyQuotaer struct{}
var _ quotaer = &dummyQuotaer{}
func newDummyQuotaer() *dummyQuotaer {
return &dummyQuotaer{}
}
func (q *dummyQuotaer) AddProject(_, _ string) (string, uint16, error) {
return "", 0, nil
}
func (q *dummyQuotaer) RemoveProject(_ string, _ uint16) error {
return nil
}
func (q *dummyQuotaer) SetQuota(_ uint16, _, _ string) error {
return nil
}
func (q *dummyQuotaer) UnsetQuota() error {
return nil
}
nfs/pkg/volume/util.go 0 → 100644
/*
Copyright 2016 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package volume
import (
"fmt"
"io/ioutil"
"math"
"os"
"regexp"
"strconv"
"strings"
"sync"
)
// generateID generates a unique exportID to assign an export
func generateID(mutex *sync.Mutex, ids map[uint16]bool) uint16 {
mutex.Lock()
id := uint16(1)
for ; id <= math.MaxUint16; id++ {
if _, ok := ids[id]; !ok {
break
}
}
ids[id] = true
mutex.Unlock()
return id
}
func deleteID(mutex *sync.Mutex, ids map[uint16]bool, id uint16) {
mutex.Lock()
delete(ids, id)
mutex.Unlock()
}
// getExistingIDs populates a map with existing ids found in the given config
// file using the given regexp. Regexp must have a "digits" submatch.
func getExistingIDs(config string, re *regexp.Regexp) (map[uint16]bool, error) {
ids := map[uint16]bool{}
digitsRe := "([0-9]+)"
if !strings.Contains(re.String(), digitsRe) {
return ids, fmt.Errorf("regexp %s doesn't contain digits submatch %s", re.String(), digitsRe)
}
read, err := ioutil.ReadFile(config)
if err != nil {
return ids, err
}
allMatches := re.FindAllSubmatch(read, -1)
for _, match := range allMatches {
digits := match[1]
if id, err := strconv.ParseUint(string(digits), 10, 16); err == nil {
ids[uint16(id)] = true
}
}
return ids, nil
}
func addToFile(mutex *sync.Mutex, path string, toAdd string) error {
mutex.Lock()
file, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0600)
if err != nil {
mutex.Unlock()
return err
}
defer file.Close()
if _, err = file.WriteString(toAdd); err != nil {
mutex.Unlock()
return err
}
file.Sync()
mutex.Unlock()
return nil
}
func removeFromFile(mutex *sync.Mutex, path string, toRemove string) error {
mutex.Lock()
read, err := ioutil.ReadFile(path)
if err != nil {
mutex.Unlock()
return err
}
removed := strings.Replace(string(read), toRemove, "", -1)
err = ioutil.WriteFile(path, []byte(removed), 0)
if err != nil {
mutex.Unlock()
return err
}
mutex.Unlock()
return nil
}
nfs/test/e2e/.gitignore 0 → 100644
/vendor
/src
\ No newline at end of file
nfs/test/e2e/e2e_test.go 0 → 100644
/*
Copyright 2015 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package e2e
import (
"testing"
"k8s.io/kubernetes/test/e2e/framework"
// test sources
_ "k8s.io/kubernetes/test/e2e/storage"
)
func init() {
framework.ViperizeFlags()
}
func TestE2E(t *testing.T) {
RunE2ETests(t)
}
nfs/test/e2e/go.mod 0 → 100644
module github.com/kubernetes-incubator/external-storage/nfs/test/e2e
go 1.12
nfs/test/e2e/nfs.go 0 → 100644
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package storage
import (
"fmt"
"io/ioutil"
"os/exec"
"path/filepath"
"time"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"k8s.io/api/core/v1"
apierrs "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
clientset "k8s.io/client-go/kubernetes"
"k8s.io/kubernetes/test/e2e/framework"
)
const (
manifestPath = "test/e2e/testing-manifests/"
nfsStatefulSetName = "nfs-provisioner"
nfsRBACCRName = "nfs-provisioner-runner"
nfsRBACCRBName = "run-nfs-provisioner"
nfsClaimName = "nfs"
nfsClaimSize = "1Mi"
nfsClassName = "example-nfs"
nfsWritePodName = "write-pod"
nfsReadPodName = "read-pod"
)
var _ = Describe("external-storage", func() {
f := framework.NewDefaultFramework("external-storage")
// filled in BeforeEach
var c clientset.Interface
var ns string
BeforeEach(func() {
c = f.ClientSet
ns = f.Namespace.Name
})
AfterEach(func() {
c.RbacV1().ClusterRoles().Delete(nfsRBACCRName, nil)
c.RbacV1().ClusterRoleBindings().Delete(nfsRBACCRBName, nil)
c.StorageV1().StorageClasses().Delete(nfsClassName, nil)
})
Describe("NFS external provisioner", func() {
mkpath := func(file string) string {
return filepath.Join(manifestPath, file)
}
It("should create and delete persistent volumes when deployed with yamls", func() {
nsFlag := fmt.Sprintf("--namespace=%v", ns)
By("creating nfs-provisioner RBAC")
cmd := exec.Command("bash", "-c", fmt.Sprintf("sed -i'' 's/namespace:.*/namespace: %s/g' %s", ns, mkpath("rbac.yaml")))
framework.ExpectNoError(cmd.Run())
framework.RunKubectlOrDie("create", "-f", mkpath("rbac.yaml"), nsFlag)
By("creating an nfs-provisioner statefulset")
tmpDir, err := ioutil.TempDir("", "nfs-provisioner-statefulset")
Expect(err).NotTo(HaveOccurred())
cmd = exec.Command("bash", "-c", fmt.Sprintf("sed -i'' 's|path:.*|path: %s|g' %s", tmpDir, mkpath("statefulset.yaml")))
framework.ExpectNoError(cmd.Run())
cmd = exec.Command("bash", "-c", fmt.Sprintf("sed -i'' '/-provisioner=/a \\ - \"-grace-period=10\"' %s", mkpath("statefulset.yaml")))
framework.ExpectNoError(cmd.Run())
framework.RunKubectlOrDie("create", "-f", mkpath("statefulset.yaml"), nsFlag)
ss, err := c.AppsV1().StatefulSets(ns).Get(nfsStatefulSetName, metav1.GetOptions{})
Expect(err).NotTo(HaveOccurred())
sst := framework.NewStatefulSetTester(c)
sst.WaitForRunningAndReady(*ss.Spec.Replicas, ss)
By("creating a class")
framework.RunKubectlOrDie("create", "-f", mkpath("class.yaml"))
By("checking the class")
class, err := c.StorageV1beta1().StorageClasses().Get(nfsClassName, metav1.GetOptions{})
Expect(err).NotTo(HaveOccurred())
By("creating a claim")
framework.RunKubectlOrDie("create", "-f", mkpath("claim.yaml"), nsFlag)
err = framework.WaitForPersistentVolumeClaimPhase(v1.ClaimBound, c, ns, nfsClaimName, framework.Poll, 60*time.Second)
Expect(err).NotTo(HaveOccurred())
By("checking the claim")
// Get new copy of the claim
claim, err := c.CoreV1().PersistentVolumeClaims(ns).Get(nfsClaimName, metav1.GetOptions{})
Expect(err).NotTo(HaveOccurred())
By("checking the volume")
// Get the bound PV
pv, err := c.CoreV1().PersistentVolumes().Get(claim.Spec.VolumeName, metav1.GetOptions{})
Expect(err).NotTo(HaveOccurred())
// Check sizes
expectedCapacity := resource.MustParse(nfsClaimSize)
pvCapacity := pv.Spec.Capacity[v1.ResourceName(v1.ResourceStorage)]
Expect(pvCapacity.Value()).To(Equal(expectedCapacity.Value()), "pvCapacity is not equal to expectedCapacity")
// Check PV properties
expectedAccessModes := []v1.PersistentVolumeAccessMode{v1.ReadWriteMany}
Expect(pv.Spec.AccessModes).To(Equal(expectedAccessModes))
Expect(pv.Spec.ClaimRef.Name).To(Equal(claim.ObjectMeta.Name))
Expect(pv.Spec.ClaimRef.Namespace).To(Equal(claim.ObjectMeta.Namespace))
Expect(pv.Spec.PersistentVolumeReclaimPolicy).To(Equal(*class.ReclaimPolicy))
Expect(pv.Spec.MountOptions).To(Equal(class.MountOptions))
By("creating a pod to write to the volume")
framework.RunKubectlOrDie("create", "-f", mkpath("write-pod.yaml"), nsFlag)
framework.ExpectNoError(framework.WaitForPodSuccessInNamespace(c, nfsWritePodName, ns))
framework.DeletePodOrFail(c, ns, nfsWritePodName)
By("creating a pod to read from the volume")
framework.RunKubectlOrDie("create", "-f", mkpath("read-pod.yaml"), nsFlag)
framework.ExpectNoError(framework.WaitForPodSuccessInNamespace(c, nfsReadPodName, ns))
framework.DeletePodOrFail(c, ns, nfsReadPodName)
By("scaling the nfs-provisioner statefulset down and up")
sst.Restart(ss)
By("creating a pod to read from the volume again")
framework.RunKubectlOrDie("create", "-f", mkpath("read-pod.yaml"), nsFlag)
framework.ExpectNoError(framework.WaitForPodSuccessInNamespace(c, nfsReadPodName, ns))
framework.DeletePodOrFail(c, ns, nfsReadPodName)
By("deleting the claim")
err = c.CoreV1().PersistentVolumeClaims(ns).Delete(nfsClaimName, nil)
if err != nil && !apierrs.IsNotFound(err) {
framework.Failf("Error deleting claim %q. Error: %v", claim.Name, err)
}
By("waiting for the volume to be deleted")
if pv.Spec.PersistentVolumeReclaimPolicy == v1.PersistentVolumeReclaimDelete {
framework.ExpectNoError(framework.WaitForPersistentVolumeDeleted(c, pv.Name, 5*time.Second, 60*time.Second))
}
})
})
})
nfs/test/e2e/test.sh 0 → 100755
#!/bin/bash
# Copyright 2018 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Vendoring the e2e framework is too hard. Download kubernetes source, patch
# our tests on top of it, build and run from there.
KUBE_VERSION=1.11.0
TEST_DIR=$GOPATH/src/github.com/kubernetes-incubator/external-storage/nfs/test/e2e
GOPATH=$TEST_DIR
# Download kubernetes source
if [ ! -e "$GOPATH/src/k8s.io/kubernetes" ]; then
mkdir -p $GOPATH/src/k8s.io
curl -L https://github.com/kubernetes/kubernetes/archive/v${KUBE_VERSION}.tar.gz | tar xz -C $TEST_DIR/src/k8s.io/
rm -rf $GOPATH/src/k8s.io/kubernetes
mv $GOPATH/src/k8s.io/kubernetes-$KUBE_VERSION $GOPATH/src/k8s.io/kubernetes
fi
cd $GOPATH/src/k8s.io/kubernetes
# Clean some unneeded sources
find ./test/e2e -maxdepth 1 -type d ! -name 'e2e' ! -name 'framework' ! -name 'manifest' ! -name 'common' ! -name 'generated' ! -name 'testing-manifests' ! -name 'perftype' -exec rm -r {} +
find ./test/e2e -maxdepth 1 -type f \( -name 'examples.go' -o -name 'gke_local_ssd.go' -o -name 'gke_node_pools.go' \) -delete
find ./test/e2e/testing-manifests -maxdepth 1 ! -name 'testing-manifests' ! -name 'BUILD' -exec rm -r {} +
# Copy our sources
mkdir ./test/e2e/storage
cp $TEST_DIR/nfs.go ./test/e2e/storage/
rm ./test/e2e/e2e_test.go
cp $TEST_DIR/e2e_test.go ./test/e2e/
cp -r $TEST_DIR/testing-manifests/* ./test/e2e/testing-manifests
# Build e2e.test
./build/run.sh make KUBE_BUILD_PLATFORMS=linux/amd64 WHAT=test/e2e/e2e.test &> /dev/null
# Download kubectl to _output directory
if [ ! -e "$HOME/bin/kubectl" ]; then
curl -o $HOME/bin/kubectl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x $HOME/bin/kubectl
fi
# Run tests assuming local cluster i.e. one started with hack/local-up-cluster.sh
./_output/dockerized/bin/linux/amd64/e2e.test --provider=local --ginkgo.focus=external-storage --kubeconfig=$HOME/.kube/config
nfs/test/e2e/testing-manifests 0 → 120000
../../deploy/kubernetes
\ No newline at end of file
Markdown is supported
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!