关于disk项目的相关笔记-zhanghaoran's blog

前段时间，在闲来无事和被某度的限速的逼迫下，想自己搞个网盘系统；但后续因其他原因，我们叫此项目为文件管理系统吧。

文件管理系统：
技术栈： vue+elementui，django+mysql

系统支持： 分片上传，断点续传，秒传，文件唯一性与公共性，文件流下载；支持生成用户并分配自定义磁盘大小，支持文件的属性修改等~ 【可以算是一个简陋型小网盘】

项目访问地址：http://disk.zhanghaoran.ren/

测试账号密码： bing/test123456

gitee前端源码： https://gitee.com/manbanzhen/disk_front

gitee后端源码： https://gitee.com/manbanzhen/disk_back

本文主要记录关于Django的一些笔记：

Django配置跨域

我使用的跨域三方库是：corsheaders

下载三方库
```
pip install corsheaders
```

INSTALLED_APPS = [
    ...
    'corsheaders',
]

添加中间件

# 尽量放的靠上一点
MIDDLEWARE = [
    ...
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.common.CommonMiddleware',
]

配置白名单

# 全部允许配置
CORS_ORIGIN_ALLOW_ALL = True
# 白名单配置
# CORS_ORIGIN_WHITELIST = ('*')
# 允许cookie  # 指明在跨域访问中，后端是否支持对cookie的操作
CORS_ALLOW_CREDENTIALS = True
# 允许的请求方式
CORS_ALLOW_METHODS = (
    'DELETE',
    'GET',
    'OPTIONS',
    'POST',
    'PUT',
)
# 允许的请求头
CORS_ALLOW_HEADERS = (
    'X-Accel-Redirect',
    'Content-Disposition',
    'accept-encoding',
    'AUTHORIZATION',
    'authorization',
    content-type',
    'origin',
    'user-agent',
)

参考： https://www.cnblogs.com/daidechong/p/11236133.html

`Nginx`的`X-Accel-Redirect`的下载控制

根据网上查到的来说，在后端程序做了鉴权等处理后，使用X-Accel-Redirect重定向到nginx,由nginx提供IO流，而不是后端服务提供下载IO,并且用户端看不到文件的存储路径

扩展：

X-Accel-Limit-Rate: 限制下载速度

X-Accel-Limit-Buffering: 设置此连接的代理缓存，将此设置为no将允许适用于Comet和HTTP流式应用程序的无缓冲响应。将此设置为yes将允许响应被缓存。默认yes。

X-Accel-Limit-Expires: 如果已传输过的文件被缓存下载，设置Nginx文件缓存过期时间，单位秒。默认不过期。

X-Accel-Limit-Charset: 设置文件字符集，默认utf8

代码实现：

下载视图中鉴权后将文件路径重定向给Nginx

from django.utils.http import urlquote
import os

class DownloadFileView(APIView):
    def get(self, request):
        ...
        鉴权等操作，获取到文件对象obj
        ...
        response = HttpResponse()
        response['content_type'] = "application/octet-stream"
        # 用于文件名称显示
        response['Content-Disposition'] = 'attachment; filename=' + os.path.basename(urlquote(obj['file_name'] + '.' + obj['file_ext']))
        # 将文件相对路径重定向nginx
        # 比如文件在/disk/disk_back/upload/q/abc.txt下存
        # 且nginx配置为：/disk/disk_back/upload/; 
        # 其值为： /disk/q/abc.txt  ==>> /disk 为nginx的配置路径， 详看下步
        # /disk[nginx] === /disk/disk_back/upload/[磁盘路径]
        response['X-Accel-Redirect'] = "/disk/q/abc.txt"
        return response

Nginx配置

server {
    listen 80;
    server_name 0.0.0.0;
    client_max_body_size 10240M;
    
    location / {
        ...
    }
    
    # 访问路径
    location /disk {
        internal;
        # 文件绝对路径
        alias /disk/disk_back/upload/;
    }
}

参考：https://www.zhangbj.com/p/507.html

js和python的AES加密解密

遗憾的是，两者目前还不能互相加解密；后续慢慢来

python 的AES加密解密

from Crypto.Cipher import AES
from binascii import b2a_hex, a2b_hex
def add_to_16(text):
    if len(text.encode('utf-8')) % 16:
        add = 16 - (len(text.encode('utf-8')) % 16)
    else:
        add = 0
    text = text + ('\0' * add)
    return text.encode('utf-8')
# 加密函数
def encrypt(text):
    key = 'zhanghaoran'.encode('utf-8')
    mode = AES.MODE_ECB
    text = add_to_16(text)
    cryptos = AES.new(key, mode)

    cipher_text = cryptos.encrypt(text)
    return b2a_hex(cipher_text).decode()
# 解密后，去掉补足的空格用strip() 去掉
def decrypt(text):
    key = 'zhanghaoran'.encode('utf-8')
    mode = AES.MODE_ECB
    cryptor = AES.new(key, mode)
    plain_text = cryptor.decrypt(a2b_hex(text))
    return bytes.decode(plain_text).rstrip('\0')

js的AES加密解密

const CryptoJS = require('crypto-js');  //引用AES源码js
const key = CryptoJS.enc.Utf8.parse("9999999999999999");  //十六位十六进制数作为密钥
const iv = CryptoJS.enc.Utf8.parse('9999999999999999');   //十六位十六进制数作为密钥偏移量
//解密方法
function Decrypt(word) {
    let encryptedHexStr = CryptoJS.enc.Hex.parse(word);
    let srcs = CryptoJS.enc.Base64.stringify(encryptedHexStr);
    let decrypt = CryptoJS.AES.decrypt(srcs, key, { iv: iv, mode: CryptoJS.mode.CBC, padding: CryptoJS.pad.Pkcs7 });
    let decryptedStr = decrypt.toString(CryptoJS.enc.Utf8);
    return decryptedStr.toString();
}
//加密方法
function Encrypt(word) {
    let srcs = CryptoJS.enc.Utf8.parse(word);
    let encrypted = CryptoJS.AES.encrypt(srcs, key, { iv: iv, mode: CryptoJS.mode.CBC, padding: CryptoJS.pad.Pkcs7 });
    return encrypted.ciphertext.toString().toUpperCase();
}
export default {
    Decrypt ,
    Encrypt
}
// 参考链接： https://www.jianshu.com/p/a47477e8126a

判断当前请求toke有效、过期与否

这里使用jwt来简单生成和解密token

鉴权装饰器：

# 加密token
def generate_jwt_handler(user):
    payload = {
        "iat": datetime.datetime.now().timestamp(),  # jwt的签发时间
        "exp": (datetime.datetime.now() + datetime.timedelta(days=1)).timestamp(),  # jwt自签发时间开始的生效时间
        "iss": "9999999999",
        "data": {
            "user_id": user.user_id
        }
    }
    return jwt.encode(payload=payload, key=settings.SECRET_KEY, algorithm="HS256")

# 解密token
def decode_jwt_handler(token):
    return jwt.decode(jwt=token, key=settings.SECRET_KEY, issuser="9999999999", algorithms=["HS256", ])


def required_login(func):
    @functools.wraps(func)
    def wrapper(request, *args, **kwargs):
        token = request.META.get('HTTP_AUTHORIZATION')

        # 没有token直接拒绝访问
        if token is None:
            return JsonResponse({"code": 401, "error": "Access denied"})
        try:
            # 验证token
            payload = decode_jwt_handler(token)

            # 可以将自定义的信息绑定到request上
            user_data = payload.get('data')
            request.user_id = user_data.get('user_id')
        except Exception as e:
            return JsonResponse({'code': 401, 'error': 'No permission to access resource!'})
        return func(request, *args, **kwargs)
    return wrapper

使用：

# 在需要的视图上添加
@method_decorator(required_login, name="dispatch")
class VerificationCapacity(APIView):
    pass

文件分片上传、断点续传、秒传（假）

各位看代码吧

前端：

<template>
  <div>
    <input type="file" ref="file">
    <button @click="getFileMD5">上传</button>
    <el-progress :percentage="process" status="success"></el-progress>
    <p>{{status_txt}}</p>
    <p>{{process}}%</p>
  </div>
</template>

<script>
import {upload,merge,ver} from "../../api/action";
import SparkMD5 from "spark-md5";

export default {
  name: "index",
  data(){
    return{
      chunkSize: 10240*1024,
      process: 0,
      fileMd5: null,
      currentFile: null,
      status_txt: '',
      key: ''
    }
  },
  methods:{
    getFileMD5(){
      if (this.$refs.file.files.length <= 0) {
        this.$message.warning("你还没有选中文件")
        return
      }
      this.status_txt = '正在校验'
      var self = this;
      var fileReader = new FileReader();
      var spark = new SparkMD5.ArrayBuffer();
      // 获取文件二进制数据
      this.currentFile = this.$refs.file.files[0]
      console.log(this.currentFile)
      fileReader.readAsArrayBuffer(this.currentFile);
      fileReader.onload = function (e) {
        spark.append(e.target.result);
        self.fileMd5 = spark.end();
        // console.log('md5', self.fileMd5);
        // console.log(self.currentFile.size);
        self.handleVer(self.fileMd5, self.currentFile.size, self.currentFile.fileName)
      };
    },
    handleVer(md5, file_size, fileName){
      let formData = new FormData();
      formData.append("md5", md5)
      formData.append("file_size", file_size)
      formData.append("file_name", fileName)
      ver(formData).then(res=>{
        console.log(res, '233333');
        this.key = res.key
        this.handleUpload(0)
      }).catch(error=>{
        this.$message({
          message: error,
          type: 'warning'
        });
        console.log(error, '45555');
      })
    },
    handleUpload(index){
      this.status_txt = '正在上传'
      let file = this.currentFile;
      let [fname, fext] = file.name.split('.');
      console.log("upload", fname, fext)

      let start = index*this.chunkSize;
      console.log(file.size, start)
      let p = (start / file.size).toFixed(2) * 100;
      this.process = p <= 100 ? p : 100

      if(start>file.size){
        this.handleMerge(file.name)
        return
      }

      let blob = file.slice(start, start+this.chunkSize);
      let blobName = `${fname}.${index}.${fext}`
      let blobFile = new File([blob], blobName)
    

      let formData = new FormData();
      formData.append('file', blobFile);
      formData.append('task_id', this.fileMd5);
      formData.append('chunk', index);
      formData.append('key', this.key);

      upload(formData)
      .then(res=>{
        console.log(res)
        this.handleUpload(++index)
      })
      .catch(error=>{
        console.log(error)
      })
    },
    handleMerge(name){
      console.log(name)
      // let formData = new FormData();
      // formData.append('task_id', this.fileMd5);
      // formData.append('filename', name);
      merge(this.fileMd5, name, this.key)
      .then(res=>{
        this.status_txt = '上传成功'
        console.log(res)
        this.$refs.file.value = ''
        this.process = 0
        this.status_txt = ''
        this.$emit('uploadSuccess')
      })
    },
  }
}
</script>

后端

@method_decorator(required_login, name="dispatch")
class VerificationCapacity(APIView):
    def post(self, request):
        try:
            file_size = request.POST.get('file_size')
            file_md5 = request.POST.get('md5')
            if not file_size or not file_md5:
                return Response({'code': 400, 'error': '参数有误'})
            user_id = request.user_id
            # 当该用户没有磁盘容量时，直接返回
            user_profile = UserProfile.objects.get(user_id=user_id)
            free_size = int(user_profile.disk_size) - int(user_profile.use_disk_size)

            if (int(file_size) > free_size):
                return Response({'code': 400, 'error': '磁盘容量不足~'})

            # 判断是否上传过此文件，如果有，不让上传了，在回收站里的话自动恢复
            file_obj = models.FileProfile.objects.filter(file_md5=file_md5, user_id=user_id)
            if file_obj:
                if file_obj.live_status == 'delete':
                    file_obj.live_status = 'normal'
                    file_obj.save()
                response_file = serializers.FileInfoSerializer(file_obj, many=True).data
                return Response({'code': 300, 'data': response_file})

            # 生成上传标识， 如果有上传过，但没上传完 接着继续上传
            encrypt_data = encrypt(str({"file_md5": file_md5, "save_path": user_profile.file_path}))
            # todo: 获取当前上传标识下有无 上传过的分片， 有则返回切片index,继续上传， 无则返回index=0，从头来
            file_path = './upload/%s/temp/%s' % (user_profile.file_path, file_md5)
            print(file_path)
            index = 0
            if (os.path.isdir(file_path)):
                while True:
                    filename = './upload/%s/temp/%s/%s%d'%(user_profile.file_path, file_md5, file_md5, index)
                    print(os.path.isfile(filename))
                    if os.path.isfile(filename):
                        index += 1
                        print('true')
                    else:
                        break
            return Response({'code': 0, 'data': {'key': encrypt_data, "index": index } })
        except Exception as e:
            return Response({'code': 500, 'error': str(e)})


@method_decorator(required_login, name="dispatch")
class SliceUploadView(APIView):
    def post(self, request):
        try:
            if not request.POST['key']:
                return Response({'code': 400, 'data': '来路不明'})
            key = request.POST['key']
            file_info = demjson.decode(decrypt(key))

            save_path = file_info['save_path']
            task = file_info['file_md5']

            upload_file = request.FILES.get('file')
            # task = request.POST.get('task_id')  # 获取文件唯一标识符
            chunk = request.POST.get('chunk', 0)  # 获取该分片在所有分片中的序号
            filename = '%s%s' % (task, chunk)  # 构成该分片唯一标识符
            print("filename=", filename)

            user_id = request.user_id
            print('user_id', user_id)
            file_obj = models.FileProfile.objects.filter(file_md5=task, user=user_id)
            print(file_obj, len(file_obj), type(file_obj))
            if len(file_obj) <= 0:
                default_storage.save('./upload/%s/temp/%s/%s' % (save_path, task, filename), ContentFile(upload_file.read()))  # 保存分片到本地
                return Response({'code': 0, 'data': 'post'})
            else:
                response_file = serializers.FileInfoSerializer(file_obj, many=True).data
                return Response({'code': 300, 'data': response_file})
        except Exception as e:
            return Response({'code': 400, 'error': str(e)})


@method_decorator(required_login, name="dispatch")
class MergeUploadView(APIView):
    def post(self, request):
        try:
            # todo: 文件安全性问题，如果上传的是一个攻击脚本呢？
            if not request.POST['key']:
                return Response({'code': 400, 'data': '来路不明'})
            key = request.POST['key']
            print('key', key)
            file_info = demjson.decode(decrypt(key))
            print(file_info, type(file_info))
            save_path = file_info['save_path']
            task = file_info['file_md5']

            # task = request.GET.get('task_id')
            current_file_name = request.POST.get('filename', '')
            en_file_name = result = p.get_pinyin(current_file_name)
            # upload_type = request.GET.get('type')
            # if len(ext) == 0 and upload_type:
            #     ext = upload_type.split('/')[1]
            ext = '' if len(current_file_name) == 0 else '.%s' % en_file_name  # 构建文件后缀名
            file_path = './upload/%s/%s%s' % (save_path, task, ext)
            chunk = 0
            print('file_path', file_path)
            with open(file_path, 'wb') as target_file:  # 创建新文件
                while True:
                    try:
                        filename = './upload/%s/temp/%s/%s%d' % (save_path, task, task, chunk)
                        source_file = open(filename, 'rb')  # 按序打开每个分片
                        target_file.write(source_file.read())  # 读取分片内容写入新文件
                        source_file.close()
                    except IOError:
                        break
                    chunk += 1
                    os.remove(filename)  # 删除该分片，节约空间
            file_size = os.path.getsize(file_path)
            file_ext = ext.split('.')[-1]
            os.removedirs('./upload/%s/temp/%s/' % (save_path, task))

            user = User.objects.get(id=request.user_id)
            user_pro = UserProfile.objects.get(user=user)
            user_pro.use_disk_size = int(user_pro.use_disk_size) + int(file_size)
            user_pro.save()
            save_file_name = current_file_name.rsplit('.', 1)[0]
            models.FileProfile(user=user, file=file_path, file_name=save_file_name, file_md5=task, file_size=file_size, file_ext=file_ext).save()
            # models.FileProfile.create(user=user, file=file_path, file_name=current_file_name, file_md5=task, file_size=file_size, file_ext=file_ext)
            # models.FileProfile.objects.update_or_create(defaults={
            #     user=user, file=file_path, file_name=current_file_name, file_md5=task, file_size=file_size, file_ext=file_ext
            # }, user=user,file_md5=task)
            return Response({'code': 0})
        except Exception as e:
            print(e)
            return Response({'code': 500, 'error': str(e)})

想提桶！！！

Django配置跨域

Nginx的X-Accel-Redirect的下载控制

js和python的AES加密解密

判断当前请求toke有效、过期与否

文件分片上传、断点续传、秒传（假）

`Nginx`的`X-Accel-Redirect`的下载控制